Dimensionality Reduction Techniques (DRT)

Transcription

1 Dimensionality Reduction Techniques (DRT) Introduction: Sometimes we have lot of variables in the data for analysis which create multidimensional matrix. To simplify calculation and to get appropriate, meaningful and valid results, it is useful to reduce dimension of the dataset. Conventional dimensionality reduction methods contain principal component analysis, factor analysis, multidimensional scaling and subset selection. In this lecture note, these methods are briefly described. 1. Principal Component Analysis This method is based on multidimensional space projection into the space with lower dimension. The aim is to find real dimension of the data. In PCA, the variables under study are transformed into a new set of variables, which are linear combination of the original variables. These new set of variables are called principal components and obtained in such a manner that the first principal component accounts for as much as possible of the variation in the original data then the second principal component and so on. It is quite likely that first few principal components account for most of the variability in the original data. If so, these few principal components can then replace the initial p variables in subsequent analysis, thus, reducing the effective dimensionality of the problem. The principal components are sensitive to the scale of measurement. The conventional way of getting rid off this problem is to use standardized variables with unit variances, i.e., correlation matrix in place of dispersion matrix. Steps to performing principal components analysis Get the data Standardized the data Calculate correlation matrix Calculate eigen values and eigen vectors of correlation matrix Choosing components and forming a feature vectors Eigen vector with the highest eigen values is principal components. Deriving the new data set Once we have chosen the components (eigenvectors) that we wish to keep in our data and formed a feature vector, we simply take the transpose of the vector and multiply it on the left of the original data set, transposed. 2. Factor Analysis Factor analysis is a method for investigating whether a number of variables of interest Y 1, Y 2, :::, Y n, are linearly related to a smaller number of unobservable factors F 1, F 2, :::, F k. However these factors are not observable. Suppose variables can be grouped by their correlations. That is, all variables within a particular group are highly correlated among themselves but have relatively small correlations with variables in a different group. It is possible that each group of variables represents a single underlying construct, or factor,

2 that is responsible for the observed correlations. There are basically two methods of factor analysis available in literatures. Exploratory factor analysis (EFA): It attempts to discover the nature of the constructs, influencing a set of responses. Confirmatory factor analysis (CFA): It tests whether a specified set of constructs is influencing responses in a predicted way. Steps to performing exploratory factor analysis Collect measurements Obtained the correlation matrix Select the number of factors for inclusion There are certain criteria for selection of optimal number of factors. Kaiser criterion states that a number of factors should be equal to the number of the eigenvalues of the correlation matrix that are greater than one. The Scree test states that you should plot the eigenvalues of the correlation matrix in descending order, and then use a number of factors equal to the number of eigenvalues that occur prior to the last major drop in eigenvalue magnitude. Extract initial set of factors There are a number of different extraction methods, including maximum likelihood, principal component and principal axis extraction. The best method is generally maximum likelihood extraction, unless we seriously lack multivariate normality in our measures. Rotate the factors to a final solution Rotation of the factors is an attempt to find a factor solution that is equal to that obtained in the initial extraction but which has the simplest interpretation. Interpret factor structure Each of our measures will be linearly related to each of our factors. The strength of this relationship is contained in the respective factor loading, produced by our rotation. This loading can be interpreted as a standardized regression coefficient, regressing the factor on the measures. Construct factor scores for further analysis If we wish to perform additional analyses using the factors as variables we will need to construct factor scores. The score for a given factor is a linear combination of all of the measures, weighted by the corresponding factor loading. Steps to performing confirmatory factor analysis Define the factor model This involves selecting the number of factors, and fining de the nature of the loadings between the factors and the measures. Collect measurements 168

3 Obtain the correlation matrix Fit the model to the data The most common model-fitting procedure is Maximum likelihood estimation, which should probably be used unless our measures seriously lack multivariate normality. Evaluate model adequacy When the factor model is fit to the data, the factor loadings are chosen to minimize the discrepancy between the correlation matrix implied by the model and the actual observation. The amount of discrepency after the best parameters are chosen can be used as a measure of how consistent the model is with the data. The most commonly used test of model adequacy is the χ2 goodness-of-fit test. Compare with other models If we want to compare two models, one of which is a reduced form of the other, we can just examine the difference between their χ2 statistics, which will also have an approximately χ2 distribution. Almost all tests of individual factor loadings can be made as comparisons of full and reduced factor models. In cases where we are not examining full and reduced models we can compare the Root mean square error of approximation (RMSEA), which is an estimate of discrepancy per degree of freedom in the model. 3. Multidimensional Scaling It is a technique for the analysis of proximity data on a set of stimuli to reveal the hidden structure underlying the data. Proximity data can come from any pairwise similarity matrix. It is widely used as an exploratory data analysis technique, by placing objects as points in low dimensional space. By a representation of the patterns of proximity in two or three dimensions, researchers can visually study the structure in the data. Steps to performing multidimensional scaling Formulating the problem Here, we define the problems by defining the purpose of study, type and number of variables to be compared etc. Obtaining input data Performing the MDS analysis Depending upon the type of data, MDS analysis is of two types, MDS analysis of interval or ratio data and MDS analysis of ordinal data. Mapping the results and defining the dimensions The statistical program (or a related module) will map the results. The map will plot each product (usually in two dimensional space). The proximity of products to each other indicate either how similar they are. Test the results for reliability and validity Compute R-squared to determine what proportion of variance of the scaled data can be accounted for by the MDS procedure. An R-square of 0.6 is considered the 169

4 minimum acceptable level. An R-square of 0.8 is considered good for metric scaling and 0.9 is considered good for non-metric scaling. Other possible tests are Kruskal s Stress, split data tests, data stability tests (i.e., eliminating one brand), and test-retest reliability. Report the results comprehensively Along with the mapping, at least distance measure (e.g. Sorenson index, Jaccard index) and reliability (e.g., stress value) should be given. 4. Subset Selection A frequent problem in data mining is that of using a regression equation to predict the value of a dependent variable when we have a number of variables available to choose as independent variables in our model. Sometimes taking all the independent variables for analysis is not very useful because of the following reasons- It may be expensive to collect the full complement of variables for future predictions. We may be able to more accurately measure fewer variables (for example in surveys). Parsimony is an important property of good models. We obtain more insight into the influence of regressors in models with a few parameters. Estimates of regression coefficients are likely to be unstable due to multicollinearity in models with many variables. It can be shown that using independent variables that are uncorrelated with the dependent variable will increase the variance of predictions. It can be shown that dropping independent variables that have small (non-zero) coefficients can reduce the average error of predictions. Algorithms of subset selection 1) Forward selection Steps of forward selection Start with constant term only in subset S. Compute the reduction in the sum of squares of the residuals (SSR) obtained by including each variable that is not presently in S. We denote by SSR(S) the sum of square residuals given that the model consists of the set S of variables. Let σ 2(S) be an unbiased estimate for σ for the model consisting of the set S of variables. For the variable, say, i, that gives the largest reduction in SSR compute If Fi > Fin, where Fin is a threshold (typically between 2 and 4) add i to S Repeat 2 until no variables can be added. 170

5 2) Backward elimination Steps of backward elimination Start with all variables in S. Compute the increase in the sum of squares of the residuals (SSR) obtained by excluding each variable that is presently in S. For the variable, say, i, that gives the smallest increase in SSR compute If Fi < Fout, where Fout is a threshold (typically between 2 and 4) then drop i from S. Repeat 2 until no variable can be dropped. 3) Step-wise Regression This procedure is like Forward Selection except that at each step we consider dropping variables as in Backward Elimination. 171