Data Mining. Dimensionality reduction. Hamid Beigy. Sharif University of Technology. Fall 1395

Transcription

1 Data Mining Dimensionality reduction Hamid Beigy Sharif University of Technology Fall 1395 Hamid Beigy (Sharif University of Technology) Data Mining Fall / 42

2 Outline 1 Introduction 2 Feature selection methods 3 Feature extraction methods Principal component analysis Kernel principal component analysis Factor analysis Locally Linear Embedding Linear discriminant analysis Hamid Beigy (Sharif University of Technology) Data Mining Fall / 42

3 Table of contents 1 Introduction 2 Feature selection methods 3 Feature extraction methods Principal component analysis Kernel principal component analysis Factor analysis Locally Linear Embedding Linear discriminant analysis Hamid Beigy (Sharif University of Technology) Data Mining Fall / 42

4 Introduction The complexity of any classifier or regressors depends on the number of input variables or features. These complexities include Time complexity: In most learning algorithms, the time complexity depends on the number of input dimensions(d) as well as on the size of training set (N). Decreasing D decreases the time complexity of algorithm for both training and testing phases. Space complexity: Decreasing D also decreases the memory amount needed for training and testing phases. Samples complexity: Usually the number of training examples (N) is a function of length of feature vectors (D). Hence, decreasing the number of features also decreases the number of training examples. Usually the number of training pattern must be 10 to 20 times of the number of features. Hamid Beigy (Sharif University of Technology) Data Mining Fall / 42

5 Introduction There are several reasons why we are interested in reducing dimensionality as a separate preprocessing step. Decreasing the time complexity of classifiers or regressors. Decreasing the cost of extracting/producing unnecessary features. Simpler models are more robust on small data sets. Simpler models have less variance and thus are less depending on noise and outliers. Description of classifier or regressors is simpler / shorter. Visualization of data is simpler. Hamid Beigy (Sharif University of Technology) Data Mining Fall / 42

6 Peaking phenomenon In practice, for a finite number of training examples (N), by increasing the number of features (l) we obtain an initial improvement in performance, but after a critical value, further increase of the number of features results in an increase of the probability of error. This phenomenon is also known as the peaking phenomenon. If the number of samples increases (N 2 N 1 ), the peaking phenomenon occures for larger number of features (l 2 > l 1 ). Hamid Beigy (Sharif University of Technology) Data Mining Fall / 42

7 Dimensionality reduction methods There are two main methods for reducing the dimensionality of inputs Feature selection: These methods select d (d < D) dimensions out of D dimensions and D d other dimensions are discarded. Feature extraction: Find a new set of d (d < D) dimensions that are combinations of the original dimensions. Hamid Beigy (Sharif University of Technology) Data Mining Fall / 42

8 Table of contents 1 Introduction 2 Feature selection methods 3 Feature extraction methods Principal component analysis Kernel principal component analysis Factor analysis Locally Linear Embedding Linear discriminant analysis Hamid Beigy (Sharif University of Technology) Data Mining Fall / 42

9 Feature selection methods Feature selection methods select d (d < D) dimensions out of D dimensions and D d other dimensions are discarded. Reasons for performing feature selection Increasing the predictive accuracy of classifiers or regressors. Removing irrelevant features. Enhancing learning efficiency (reducing computational and storage requirements). Reducing the cost of future data collection (making measurements on only those variables relevant for discrimination/prediction). Reducing complexity of the resulting classifiers/regressors description (providing an improved understanding of the data and the model). Feature selection is not necessarily required as a pre-processing step for classification /regression algorithms to perform well. Several algorithms employ regularization techniques to handle over-fitting or averaging such as ensemble methods. Hamid Beigy (Sharif University of Technology) Data Mining Fall / 42

10 Feature selection methods Feature selection methods can be categorized into three categories. Filter methods: These methods use the statistical properties of features to filter out poorly informative features. Wrapper methods: These methods evaluate the feature subset within classifier/regressor algorithms. These methods are classifier/regressors dependent and have better performance than filter methods. Embedded methods:these methods use the search for the optimal subset into classifier/regression design. These methods are classifier/regressors dependent. Two key steps in feature selection process. Evaluation: An evaluation measure is a means of assessing a candidate feature subset. Subset generation: A subset generation method is a means of generating a subset for evaluation. Hamid Beigy (Sharif University of Technology) Data Mining Fall / 42

11 Feature selection methods (Evaluation measures) Large number of features are not informative- either irrelevant or redundant. Irrelevant features are those features that don t contribute to a classification or regression rule. Redundant features are those features that are strongly correlated. In order to choose a good feature set, we require a means of a measure to contribute to the separation of classes, either individually or in the context of already selected features. We need to measure relevancy and redundancy. There are two types of measures Measures that relay on the general properties of the data. These assess the relevancy of individual features and are used to eliminate feature redundancy. All these measures are independent of the final classifier. These measures are inexpensive to implement but may not well detect the redundancy. Measures that use a classification rule as a part of their evaluation. In this approach, a classifier is designed using the reduced feature set and a measure of classifier performance is employed to assess the selected features. A widely used measure is the error rate. Hamid Beigy (Sharif University of Technology) Data Mining Fall / 42

12 Feature selection methods (Evaluation measures) The following measures relay on the general properties of the data. Feature ranking: Features are ranked by a metric and those that fail to achieve a prescribed score are eliminated. Examples of these metrics are: Pearson correlation, mutual information, and information gain. Interclass distance: A measure of distance between classes is defined based on distances between members of each class. Example of these metrics is: Euclidean distance. Probabilistic distance: This is the computation of a probabilistic distance between class-conditional probability density functions, i.e. the distance between p(x C 1 ) and p(x C 2 ) (two-classes). Example of these metrics is: Chhernoff dissimilarity measure. Probabilistic dependency: These measures are multi-class criteria that measure the distance between class-conditional probability density functions and the mixture probability density function for the data irrespective of the class, i.e. the distance between p(x C i ) and p(x). Example of these metrics is: Joshi dissimilarity measure. Hamid Beigy (Sharif University of Technology) Data Mining Fall / 42

13 Feature selection methods (Search algorithms) Complete search: These methods guarantee to find the optimal subset of features according to some specified evaluation criteria. For example exhaustive search and branch and bound methods are complete. Sequential search: In these methods, features are added or removed sequentially. These methods are not optimal, but are simple to implement and fast to produce results. Best individual N: The simplest method is to assign a score to each feature and then select N top ranks features. Sequential forward selection: It is a bottom-up search procedure that adds new features to a feature set one at a time until the final feature set is reached. Generalized sequential forward selection: In this approach, at each time r > 1, features are added instead of a single feature. Sequential backward elimination: It is a top-down procedure that deletes a single feature at a time until the final feature set is reached. Generalized sequential backward elimination : In this approach, at each time r > 1 features are deleted instead of a single feature. Hamid Beigy (Sharif University of Technology) Data Mining Fall / 42

14 Table of contents 1 Introduction 2 Feature selection methods 3 Feature extraction methods Principal component analysis Kernel principal component analysis Factor analysis Locally Linear Embedding Linear discriminant analysis Hamid Beigy (Sharif University of Technology) Data Mining Fall / 42

15 Feature extraction In feature extraction, we are interested to find a new set of k (k > D) dimensions that are combinations of the original D dimensions. Feature extraction methods may be supervised or unsupervised. Examples of feature extraction methods Principal component analysis (PCA) Factor analysis (FA)s Multi-dimensional scaling (MDS) ISOMap Locally linear embedding Linear discriminant analysis (LDA) Hamid Beigy (Sharif University of Technology) Data Mining Fall / 42

16 Principal component analysis PCA project D dimensional input vectors to k dimensional input vectors via a linear mapping with minimum loss of information. Dimensions are combinations of the original D dimensions. The problem is to find a matrix W such that the following mapping results in the minimum loss of information. Z = W T X PCA is unsupervised and tries to maximize the variance. The principle component is w 1 such that the sample after projection onto w 1 is most spread out so that the difference between the sample points becomes most apparent. For uniqueness of the solution, we require w 1 = 1, Let Σ = Cov(X ) and consider the principle component w 1, we have z 1 = w T 1 x Var(z 1 ) = E[(w T 1 x w T 1 µ) 2 ] = E[(w T 1 x w T 1 µ)(w T 1 x w T 1 µ)] = E[w T 1 (x µ)(x µ) T w 1 ] = w T 1 E[(x µ)(x µ) T ]w 1 = w T 1 Σw 1 Hamid Beigy (Sharif University of Technology) Data Mining Fall / 42

17 Principal component analysis (cont.) The mapping problem becomes w 1 = argmax w w T Σw subject to w T 1 w 1 = 1. Writing this as Lagrange problem, we have maximize w 1 w T 1 Σw 1 α(w T 1 w 1 1) Taking derivative with respect to w 1 and setting it equal to 0, we obtain 2Σw 1 = 2αw 1 Σw 1 = αw 1 Hence w 1 is eigenvector of Σ and α is the corresponding eigenvalue. Since we want to maximize Var(z 1 ), we have Var(z 1 ) = w1 T Σw 1 = αw1 T w 1 = α Hence, we choose the eigenvector with the largest eigenvalue, i.e. λ 1 = α. Hamid Beigy (Sharif University of Technology) Data Mining Fall / 42

18 Principal component analysis (cont.) The second principal component, w 2, should also maximize variance be unit length orthogonal to w 1 (z 1 and z 2 must be uncorrelated) The mapping problem for the second principal component becomes w 2 = argmax w w T Σw subject to w T 2 w 2 = 1 and w T 2 w 1 = 0. Writing this as Lagrange problem, we have maximize w 2 w T 2 Σw 2 α(w T 2 w 2 1) β(w T 2 w 1 0) Taking derivative with respect to w 2 and setting it equal to 0, we obtain 2Σw 2 2αw 2 βw 1 = 0 Pre-multiply by w T 1, we obtain 2w T 1 Σw 2 2αw T 1 w 2 βw T 1 w 1 = 0 Note that w T 1 w 2 = 0 and w T 1 Σw 2 = (w T 2 Σw 1) T = w T 2 Σw 1 is a scaler. Hamid Beigy (Sharif University of Technology) Data Mining Fall / 42

19 Principal component analysis (cont.) Since Σw 1 = λ 1 w 1, therefore we have Then β = 0 and the problem reduces to w T 1 Σw 2 = w T 2 Σw 1 = λ 1 w T 2 w 1 = 0 Σw 2 = αw 2 This implies that w 2 should be the eigenvector of Σ with the second largest eigenvalue λ 2 = α. Similarly, you can show that the other dimensions are given by the eigenvectors with decreasing eigenvalues. Since Σ is symmetric, for two different eigenvalues, their corresponding eigenvectors are orthogonal. (Show it) If Σ is positive definite (x T Σx > 0 for all non-null vector x), then all its eigenvalues are positive. If Σ is singular, its rank is k (k < D) and λ i = 0 for i = k + 1,..., D. Hamid Beigy (Sharif University of Technology) Data Mining Fall / 42

20 Principal component analysis (cont.) Define Z = W T (X m) Then k columns of W are the k leading eigenvectors of S (the estimator of Σ). m is the sample mean of X. Subtracting m from X before projection centers the data on the origin. How to normalize variances? Hamid Beigy (Sharif University of Technology) Data Mining Fall / 42

21 Principal component analysis (cont.) How to select k? If all eigenvalues are positive and S = D i=1 λ i is small, then some eigenvalues have little contribution to the variance and may be discarded. Scree graph is the plot of variance as a function of the number of eigenvectors. Hamid Beigy (Sharif University of Technology) Data Mining Fall / 42

22 Principal component analysis (cont.) How to select k? We select the leading k components that explain more than for example 95% of the variance. The proportion of variance (POV) is POV = k i=1 λ i D i=1 λ i By visually analyzing it, we can choose k. Hamid Beigy (Sharif University of Technology) Data Mining Fall / 42

23 Principal component analysis (cont.) How to select k? Another possibility is to ignore the eigenvectors whose corresponding eigenvalues are less than the average input variance (why?). In the pre-processing phase, it is better to pre-process data such that each dimension has mean 0 and unit variance(why and when?). Question: Can we use the correlation matrix instead of covariance matrix? Drive solution for PCA. Hamid Beigy (Sharif University of Technology) Data Mining Fall / 42

24 Principal component analysis (cont.) PCA is sensitive to outliers. A few points far from the center have large effect on the variances and thus eigenvectors. Question: How can use the robust estimation methods for calculating parameters in the presence of outliers? A simple method is discarding the isolated data points that are far away. Question: When D is large, calculating, sorting, and processing of S may be tedious. Is it possible to calculate eigenvectors and eigenvalues directly from data without explicitly calculating the covariance matrix? Hamid Beigy (Sharif University of Technology) Data Mining Fall / 42

25 Principal component analysis (reconstruction error) In PCA, input vector x is projected to the z space as z = W T (x µ) When W is an orthogonal matrix (W T W = I ), it can be back-projected to the original space as ˆx = Wz + µ ˆx is the reconstruction of x from its representation in the z space. Question: Show that PCA minimizes the following reconstruction error. ˆx i x i 2 i The reconstruction error depends on how many of the leading components are taken into account. Hamid Beigy (Sharif University of Technology) Data Mining Fall / 42

26 Kernel principal component analysis PCA can be extended to find nonlinear directions in the data using kernel methods. Kernel PCA finds the directions of most variance in the feature space instead of the input space. Linear principal components in the feature space correspond to nonlinear directions in the input space. Using kernel trick, all operations can be carried out in terms of the kernel function in input space without having to transform the data into feature space. Let φ correspond to a mapping from the input space to the feature space. Each point in feature space is given as the image of φ(x) of the point x in the input space. In feature space, we can find the first kernek principal component W 1 (W T 1 W 1 = 1) by solving Σ φ W 1 = λ 1 W 1 Covariance matrix Σ φ in feature space is equal to Σ φ = 1 N We assume that the points are centered. N φ(x i )φ(x i ) T i=1 Hamid Beigy (Sharif University of Technology) Data Mining Fall / 42

27 Kernel principal component analysis (cont.) Plugging Σ φ into Σ φ W 1 = λ 1 W 1, we obtain ( ) 1 N φ(x i )φ(x i ) T W 1 = λ 1 W 1 N 1 N i=1 N ) φ(x i ) (φ(x i ) T W 1 i=1 i=1 = λ 1 W 1 N ( φ(xi ) T ) W 1 φ(x i ) = W 1 Nλ 1 N c i φ(x i ) = W 1 i=1 c i = φ(x i ) T W 1 Nλ 1 is a scalar value Hamid Beigy (Sharif University of Technology) Data Mining Fall / 42

28 Kernel principal component analysis (cont.) Now substitute N i=1 c iφ(x i ) = W 1 in Σ φ W 1 = λ 1 W 1, we obtain ( ) ( 1 N N ) φ(x i )φ(x i ) T c i φ(x i ) N i=1 i=1 N = λ 1 c i φ(x i ) i=1 1 N N N c j φ(x i )φ(x i ) T φ(x j ) = λ 1 c i φ(x i ) N i=1 j=1 i=1 N N N φ(x i ) c j φ(x i ) T φ(x j ) = Nλ 1 c i φ(x i ) i=1 j=1 i=1 Replacing φ(x i ) T φ(x j ) by K(x i, x j ) N N N φ(x i ) c j K(x i, x j ) = Nλ 1 c i φ(x i ) i=1 j=1 i=1 Hamid Beigy (Sharif University of Technology) Data Mining Fall / 42

29 Kernel principal component analysis (cont.) Multiplying with φ(x k ) T, we obtain N N N φ(x k ) T φ(x i ) c j K(x i, x j ) = Nλ 1 c i φ(x k ) T φ(x i ) i=1 j=1 i=1 N N N K(x k, x i ) c j K(x i, x j ) = Nλ 1 c i K(x k, x j ) i=1 j=1 i=1 By some algebraic simplificatin, we obtain (do it) Multiplying by K 1, we obtain K 2 C = Nλ 1 KC KC = Nλ 1 C KC = η 1 C Weight vector C is the eigenvector corresponding to the largest eigenvalue η 1 of the kernel matrix K. Hamid Beigy (Sharif University of Technology) Data Mining Fall / 42

30 Kernel principal component analysis (cont.) Replacing N i=1 c iφ(x i ) = W 1 in constraint W T 1 W 1 = 1, we obtain N i=1 j=1 N c j c j φ(x i ) T φ(x j ) = 1 C T KC = 1 Using KC = η 1 C, we obtain C T (η 1 C) = = 1 η 1 C T C = = 1 C 2 = 1 η 1 Since C is an eigenvector of K, it will have unit norm. To ensure that W 1 is a unit vector, multiply C by 1 η1 Hamid Beigy (Sharif University of Technology) Data Mining Fall / 42

31 Kernel principal component analysis (cont.) In general, we do not map input space to the feature space via φ, hence we cannot compute W 1 using N c i φ(x i ) = W 1 i=1 We can project any point φ(x) on to principal direction W 1 W T 1 φ(x) = N c i φ(x i ) T φ(x) = i=1 N c i K(x i, x) i=1 When x = x i is one of the input points, we have a i = W1 T φ(x i ) = Ki T C where K i is the column vector corresponding to the I th row of K and a i is the vector in the reduced dimension. This shows that all computation are carried out using only kernel operations. Hamid Beigy (Sharif University of Technology) Data Mining Fall / 42

32 Factor analysis In PCA, from the original dimensions x i (for i = 1,..., D), we form a new set of variables z that are linear combinations of x i Z = W T (X µ) In factor analysis (FA), we assume that there is a set of unobservable, latent factors z j (for j = 1,..., k), which when acting in combination generate x. Thus the direction is opposite that of PCA. The goal is to characterize the dependency among the observed variables by means of a smaller number of factors. Suppose there is a group of variables that have high correlation among themselves and low correlation with all the other variables. Then there may be a single underlying factor that gave rise to these variables. FA, like PCA, is a one-group procedure and is unsupervised. The aim is to model the data in a smaller dimensional space without loss of information. In FA, this is measured as the correlation between variables. Hamid Beigy (Sharif University of Technology) Data Mining Fall / 42

33 Factor analysis (cont.) Hamid Beigy (Sharif University of Technology) Data Mining Fall / 42

34 Factor analysis (cont.) As in PCA, we have a sample x drawn from some unknown probability density with E[x] = µ and Cov(x) = Σ. We assume that the factors are unit normals, E[z j ] = 0, Var(z j ) = 1, and are uncorrelated,cov(z i, z j ) = 0, i j. To explain what is not explained by the factors, there is an added source for each input which we denote by ɛ i. It is assumed to be zero-mean, E[ɛ i ] = 0, and have some unknown variance, Var(ɛ i ) = ψ i. These specific sources are uncorrelated among themselves, Cov(ɛ i, ɛ i ) = 0, i j, and are also uncorrelated with the factors, Cov(ɛ i, z j ) = 0, i, j. FA assumes that each input dimension, x i can be written as a weighted sum of k < D factors, z j plus the residual term. This can be written in vector-matrix form as x i µ i = v i1 z 1 + v i2 z v ik z k + ɛ i x µ = VZ + ɛ i V is the D k matrix of weights, called factor loadings. Hamid Beigy (Sharif University of Technology) Data Mining Fall / 42

35 Factor analysis (cont.) We assume that µ = 0 and we can always add µ after projection. Given that Var(z j ) = 1 and Var(ɛ i ) = ψ i and In vector-matrix form, we have Var(x i ) = v 2 i1 + v 2 i v 2 ik + ψ i Σ = Var(X ) = VV T + Ψ There are two uses of factor analysis: It can be used for knowledge extraction when we find the loadings and try to express the variables using fewer factors. It can also be used for dimensionality reduction when k < D. Hamid Beigy (Sharif University of Technology) Data Mining Fall / 42

36 Factor analysis (cont.) For dimensionality reduction, we need to find the factor, z j, from x i. We want to find the loadings w ji such that z j = D w ji x i + ɛ i i=1 In vector form, for input x, this can be written as z = W T x + ɛ For N input, the problem is reduced to linear regression and its solution is W = (X T X ) 1 X T Z We do not know Z; We multiply and divide both sides by N 1 and obtain (drive it!) W = (N 1)(X T X ) 1 X T ( Z X T ) 1 N 1 = X X T Z N 1 N 1 = S 1 V Z = XW = XS 1 V Hamid Beigy (Sharif University of Technology) Data Mining Fall / 42

37 Locally Linear Embedding Locally linear embedding (LLE) recovers global nonlinear structure from locally linear fits. The idea is that each point can be approximated as a weighted sum of its neighbors. The neighbors either defined using a given number of neighbors (n) or distance threshold (ɛ). Let x r be an example in the input space and its neighbors be x(r) s. We find weights in such a way that minimize the following objective function. E[W x] = N x r s r=1 w rs x(r) s The idea in LLE is that the reconstruction weights w rs reflect the intrinsic geometric properties of the data is also valid for the new space. The first step of LLE is to find w rs in such a way that minimize the above objective function subject to s w rs = 1. 2 Hamid Beigy (Sharif University of Technology) Data Mining Fall / 42

38 Locally Linear Embedding (cont.) traints: first, that ructed only from 0 if X j does ij The second step of LLE is to keep w rs fixed and construct the new coordinates Y in such a way that minimize the following objective function. N E[Y W ] = y r 2 R EPORTS w rs y(r) s not belong to the set of neighbors of Xi ; W ij subject to these constraints s (6) are found second, that the rows of the weight matrix sum to one: in such a way j W that ij 1. The optimal weights Cov(Y ) = I and E[Y ] = 0. r=1 by solving a least-squares problem (7). The constrained weights that minimize these reconstruction errors obey an important symmetry: for any particular data point, they are invariant to rotations, rescalings, and translations of that data point and its neighbors. By symmetry, it follows that the reconstruction weights characterize intrinsic geometric properties of each neighborhood, as opposed to properties that depend on a particular frame of reference (8). Note that the invariance to translations is specifically enforced by the sum-to-one constraint on the rows of the weight matrix. Suppose the data lie on or near a smooth nonlinear manifold of lower dimensionality d D. To a good approximation then, there exists a linear mapping consisting of a translation, rotation, and rescaling that maps the high-dimensional coordinates of each neighborhood to global internal coordinates on the manifold. By design, the reconstruction weights W ij reflect intrinsic geometric properties of the data that are invariant to exactly such transformations. We therefore expect their characterization of local geometry in the original data space to be equally valid for local patches on the manifold. In particular, the same weights W ij that recon- S. T. Roweis and L. K. Saul, Nonlinear Dimensionality Reduction by Locally Linear Embedding, SCIENCE, VOL 290, No. 22, pp , Dec Hamid Beigy (Sharif University of Technology) Data Mining Fall / 42

39 Linear discriminant analysis Linear discriminant analysis (LDA) is a supervised method for dimensionality reduction for classification problems. Consider first the case of two classes, and suppose we take the D dimensional input vector x and project it down to one dimension using z = W T x If we place a threshold on z and classify z w 0 as class C 1, and otherwise class C 2, then we obtain our standard linear classifier. In general, the projection onto one dimension leads to a considerable loss of information, and classes that are well separated in the original D dimensional space may become strongly overlapping in one dimension. Hamid Beigy (Sharif University of Technology) Data Mining Fall / 42

40 Linear discriminant analysis(cont.) However, by adjusting the components of the weight vector W, we can select a projection that maximizes the class separation. Consider a two-class problem in which there are N 1 points of class C 1 and N 2 points of class C 2, so that the mean vectors of the two classes are given by m j = 1 N j i C j x i Hamid Beigy (Sharif University of Technology) Data Mining Fall / 42

41 Linear discriminant analysis (cont.) x 1 x 1 The simplest measure of the separation of the classes, when projected onto W, is the separation of the projected class means. This suggests that we might choose W so as to maximize where m 2 m 1 = W T (m 2 m 1 ) m j = W T m j This expression can be made arbitrarily large simply by increasing the magnitude of W. To solve this problem, we could constrain W to have unit length, so that i w i 2 = 1. Using a Lagrange multiplier to perform the constrained maximization, we then find that W (m 2 m 1 ) x 2 This axis yields better class separability ' 1 ' 2 x 1 This axis has a larger distance between means Hamid Beigy (Sharif University of Technology) Data Mining Fall / 42

42 Linear discriminant analysis (cont.) This approach has a problem: The following figure shows two classes that are well separated in the original two dimensional space but that have considerable overlap when projected onto the line joining their means. This difficulty arises from the strongly non-diagonal covariances of the class distributions. The idea proposed by Fisher is to maximize a function that will give a large separation between the projected class means while also giving a small variance within each class, thereby minimizing the class overlap. Hamid Beigy (Sharif University of Technology) Data Mining Fall / 42

43 Linear discriminant analysis (cont.) The idea proposed by Fisher is to maximize a function that will give a large separation between the projected class means while also giving a small variance within each class, thereby minimizing the class overlap. The projection z = W T x transforms the set of labeled data points in x into a labeled set in the one-dimensional space z. The within-class variance of the transformed data from class C j equals s 2 j = i C j (z i m j ) 2 where z i = w T x i. We can define the total within-class variance for the whole data set to be s s2 2.. The Fisher criterion is defined to be the ratio of the between-class variance to the within-class variance and is given by J(W ) = (m 2 m 1 ) 2 s s2 2 Hamid Beigy (Sharif University of Technology) Data Mining Fall / 42

44 Linear discriminant analysis (cont.) Between-class covariance matrix equals to S B = (m 2 m 1 )(m 2 m 1 ) T Total within-class covariance matrix equals to S W = (x i m 1 ) (x i m 1 ) T + (x i m 2 ) (x i m 2 ) T i C 1 i C 2 J(W ) can be written as J(w) = W T S B W W T S W W To maximize J(W ), we differentiate with respect to W and we obtain ( ) ( ) W T S B W S W W = W T S W W S B W S B W is always in the direction of (m 2 m 1 ) (Show it!). We can drop the scalar factors W T S B W and W T S W W (why?). Multiplying both sides of S 1 W we then obtain (Show it!) W S 1 W (m 2 m 1 ) Hamid Beigy (Sharif University of Technology) Data Mining Fall / 42

45 Linear discriminant analysis (cont.) The result W S 1 W (m 2 m 1 ) is known as Fisher s linear discriminant, although strictly it is not a discriminant but rather a specific choice of direction for projection of the data down to one dimension. The projected data can subsequently be used to construct a discriminant function. The above idea can be extended to multiple classes (Read section of Bishop). How the Fisher s linear discriminant can be used for dimensionality reduction? (Show it!) Hamid Beigy (Sharif University of Technology) Data Mining Fall / 42