A sparse factor analysis model for high dimensional latent spaces

Transcription

1 A sparse factor analysis model for high dimensional latent spaces Chuan Gao Institute of Genome Sciences & Policy Duke University Barbara E Engelhardt Department of Biostatistics & Bioinformatics Institute for Genome Sciences & Policy Duke University barbara.engelhardt@duke.edu Abstract Inducing sparsity in factor analysis has become increasingly important as applications have arisen that are best modeled by a high dimensional, sparse latent space, and the interpretability of this latent space is critical. Applying latent factor models with a high dimensional latent space but without sparsity yields nonsense factors that may be artifactual and are prone to overfittiing the data. In the Bayesian context, a number of sparsity-inducing priors have been proposed, but none that specifically address the context of a high dimensional latent space. Here we describe a Bayesian sparse factor analysis model that uses a general three parameter beta prior, which, given specific settings of hyperparameters, can recapitulate sparsity inducing priors with appropriate modeling assumptions and computational properties. We apply the model to simulated and real gene expression data sets to illustrate the model properties and to identify large numbers of sparse, possibly correlated factors in this space. Introduction Factor analysis has been used in a variety of settings to extract useful features from high dimensional data sets [, 2]. Factor analysis, in a general context, has a number of drawbacks, such as unidentifiability with respect to the rotation of the latent matrices, and the difficulty of selecting the appropriate number of factors. One solution that addresses these drawbacks is to induce sparsity in the loading matrix. By imposing substantial regularization on the loading matrix, the identifiability issue can be alleviated when the latent space is sufficiently sparse, and model selection criteria appear to be more effective at choosing the number of factors because the model does not overfit to the same extent as a non-sparse model. There are currently a number of options for how to induce sparsity constraints on the latent parameter space. We choose to work in the Bayesian context, where a sparsity-inducing prior should have substantial mass around zero to provide strong shrinkage near zero, and also have heavy tails to allow signals to escape strong shrinkage [3]. In the context of sparse regression, there have been a number of proposed solutions [4, 5, 6, 7, 8], some of which have been applied to latent factor models[9, 0]. However, all of these approaches in the factor analysis context impose an equal amount of shrinkage on all parameters, which may sacrifice small signals to achieve high levels of sparsity. To address this issue in the Bayesian latent factor model context, one can use a mixture with a point mass at zero and a normal distribution, a so-called spike and slab prior, on the loading matrix []. Unfortunately there is no closed form solution for the parameters estimates, so MCMC is used to estimate the parameters, which is computational intractable for large data. In this work, we use a three parameter beta (T PB) prior [] as a general shrinkage prior for the factor loading matrix of a latent factor model. T PB(a, b, φ) is a generalized form of the Beta distribution, with the third parameter φ further controlling the shape of the density. It has been shown that a linear transformation of the Beta distribution, producing the inverse Beta distribution, has

2 desirable shrinkage properties in sparse modeling (the horseshoe prior) [7]. The T PB distribution can be used to mimic this distribution, with the inverse beta variable scaled by φ. The T PB is thus appealing because a) it can be used to recapitulate the sparsity-inducing properties of the horseshoe prior, with substantial mass around zero to provide strong shrinkage for noise and heavy tails to avoid shrinking signal, and b) by carefully controlling its parameters, it recapitulates the two-groups model [2, 3] for priors with different shrinkage characteristics. This allows us to recreate a twogroups sparsity-inducing prior, with one mode centered at zero and the other at the posterior mean, and also has a straightforward posterior distribution for which the parameters can be estimated via expectation maximization, making it computationally tractable. In the setting of identifying a large number of factors that may individually contribute minimally to the data variance, these two features, namely computationally tractability and two-groups sparsity modeling, are critical to effectively model the data. 2 Bayesian sparse factor model via T PB We will define the factor analysis model as follows: Y = ΛX + W, () where Y has dimension n m, Λ is the loading matrix with dimension n p, X is the factor matrix with dimension p m, and W is the n m residual error matrix, where we assume W N(0, Ψ). For computational tractability, we assume Ψ is diagonal (but the diagonal elements are not necessarily the same). For the latent variable X, we follow convention by giving it a standard normal prior, X N(0, I). To induce sparsity in the factor loading matrix Λ, we put the following priors on each element λ ik of the parameter matrix Λ: λ ik N(0, ) (2) T PB(a, b, γ k ) (3) γ k T PB(c, d, ν). (4) In the prior on λ ik, ρ ij provides local shrinkage for each element, while γ k controls the global shrinkage and is specific to each factor k. As in the horseshoe, [0, ] has the desirable properties of strong shrinkage to zero while not overly shrinking signals. This general model is able to capture a number of different types of shrinkage scenarios, depending on the values of a, b, ρ (Table ). Table : Table showing the shrinkage effects for different values of a, b and ρ. ρ > < a = b = 2 horseshoe strong weak a and b weak variable weak a and b strong strong variable 3 Posterior distribution We will generalize this prior further for the latent factor model. For a given parameter θ and scale φ, the following relationship holds []: θ φ β (a, b) θ G(a, λ) and λ G(b, φ), (5) where β (a, b) and G indicate a inverse beta and gamma distribution respectively. From Equations 2, 3 and 4, if we make the substitutions θ ik = and φ k = γ k, it can be shown that θ ik φ k β (a, b). This relationship implies the following simple hierarchical structure for the latent factor model: λ ik N(0, θ ik ), θ ik G(a, δ ik ), δ ik G(b, φ k ), φ k G(c, η) and η G(d, ν), where the parameter φ k controls the global shrinkage and θ ik controls the local shrinkage. We give Ψ an uninformative prior. 2

3 Based on the posterior distributions, a Gibbs sampler can be constructed to iteratively sample the parameter values from their posterior distributions. For faster computation, we use Expectation Maximization (EM), where the expectation step involves taking the expected value of the latent variable X, and the maximization step identifies MAP parameter estimates (see paper for details). 4 Results 4. Simulations We simulated five data sets with different levels of sparsity to test the performance of our model, with sample size n = 200, m = 500, and p = 20 factors. The loading matrices Λ were generated from the above model, setting a = b =. To adjust the sparsity of the matrix, we let ν take values in the range [0 4, ], where smaller values of ν produce more sparsity in the matrix. Both X and W were chosen from N(0, I) with appropriate dimensions. We set both a = b = 2 to recapitulate the horseshoe prior and differentiate them from the simulated distribution. For ν, we used values between and 0.0 with minimal changes in the estimates. We ran EM from a random starting point ten times, and used the parameters from the run with the best fit. We compared our result with the Bayesian Factor Regression Model (BFRM) [] and the K-SVD model [9], BFRM was run with default settings, with a burn in period of 2000 and a sampling period of 20, 000, and K-SVD was run with the same setting as the demonstration file included in the package. We compared the three models by looking at the sparsity level of the three methods versus the amount of information represented in the latent subspace. The sparsity level was measured by the Gini index [3]. For an ordered list of values c, the Gini index is: 2 ( ) N c (k) N k+ 2 k= c N, where N is the total number of elements in the list; bigger values indicate a sparser representation. The accuracy of of the prediction is reflected in the mean squared error (MSE), computed from the residuals. We find that, compared to BFRM, T PB achieves equivalent or better MSE with far more sparsity (range of [0.5, 0.9] for T PB versus [0.4, 0.6] for BFRM). Compared to K-SVD, our method adaptively learned the sparsity of the data, keeping MSE low, while K-SVD maintains the same sparsity level for all simulations, sacrificing accuracy for sparsity. Interestingly, for sparser simulations, both K-SVD and T PB achieve sparser estimates than the real data, while maintaing the same prediction accuracies. We find that the Bayes Information Criterion (BIC) score, depending on the value of ν, is fairly accurate in terms of the number of selected features in this context (Figure 2). Figure : Comparison of the sparsity level (left panel, Y-axis) and the MSE (right panel, Y-axis) of T PB, BFRM and K-SVD. The underlying sparsity is on the X-axis. The true sparsity plot on the left panel also corresponds to the line with slope of. 4.2 Gene Expression Analysis Microarrays are able to generate gene expression levels for tens of thousands of genes in a sample quickly and at low cost. Biologists know that genes do not function as independent units, but instead as parts of complicated networks with different biochemical purposes [4, 5]. As a result, genes that share similar functions tend to have gene expression levels that are correlated across samples because, for example, they may be regulated by common transcription factors. Identifying these coregulated sets of genes from high dimensional gene expression measurements is critical for analysis of gene networks and for identifying genetic variants that impact transcription from long genomic distances. The number of co-regulated sets of genes may be very large, relative to the number of genes in the gene expression matrix. 3

4 Data nu=0. nu=0.0 nu=0.00 nu=0.000 LogL BIC N-factor N-factor Figure 2: Plot showing the fitness of the model, with factor number on the x-axis and log likelihood (left panel) or the BIC score (right panel) on the y-axis. In this simulated data with twenty factors, the BIC score is fairly accurate in terms of the number of selected features across different values of ν. To this end, we applied our method on a subset of 8262 genes with a sample size of 354 human cerebellum samples [unpublished]. We set K = 000 and ran EM from ten starting points with a = 0.5, b = 000 and ν = 0 4 to induce strong shrinkage; we used the result with the best fit. We note that the sparse prior alleviates the problem of overfitting by shrinking unnecessary factors to 0. By looking at the correlation of the genes that load on each factor (those that have values 0), we found that the genes on the same factors clustered well (Figure 3, left). The size of the gene clusters range from 0 to 500, with most around 50 (Figure 3, right). Figure 3: Correlation of genes loaded on the first few factors (left) and distribution of the gene clusters for a total of 000 factors (right). Factors on the left are denoted by black lines. 5 Conclusions We built a model for sparse factor analysis using a three parameter beta prior to induce shrinkage. We found that this model has favorable characteristics to estimating possibly high-dimensional latent spaces. We are further testing the robustness of estimates from our model and will use the factors to identify genetic variants that are associated with long-distance genetic regulation of each factor. Acknowledgments The authors would like to thank Sayan Mukherjee for helpful conversations. The gene expression data were generated by Merck Research Laboratories in collaboration with the Harvard Brain Tissue Resource Center and was obtained through the Synapse data repository (data set id: syn4505 at References [] C. M. Carvalho, J. E. Lucas, Q. Wang, J. Chang, J. R. Nevins, and M. West. High-dimensional sparse factor modelling - applications in gene expression genomics. Journal of the American Statistical Association, 03: , PMCID [2] Mingyuan Zhou, Lauren Hannah, David Dunson, and Lawrence Carin. Beta-Negative Binomial Process and Poisson Factor Analysis. December 20. 4

5 [3] Nicholas G. Polson James G. Scott. Shrink globally, act locally: Sparse bayesian regularization and prediction. [4] Michael E. Tipping. Sparse bayesian learning and the relevance vector machine. J. Mach. Learn. Res., :2 244, September 200. [5] Jim and Philip. Inference with normal-gamma prior distributions in regression problems. Bayesian Analysis, 5():7 88, 200. [6] Trevor Park and George Casella. The Bayesian Lasso. Journal of the American Statistical Association, 03(482):68 686, [7] Carlos M. Carvalho, Nicholas G. Polson, and James G. Scott. Handling sparsity via the horseshoe. Journal of Machine Learning Research - Proceedings Track, pages 73 80, [8] Barbara Engelhardt and Matthew Stephens. Analysis of population structure: A unifying framework and novel methods based on sparse factor analysis. PLoS Genetics, 6(9):e007, 200. [9] M. Aharon, M. Elad, and A. Bruckstein. K-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation. Signal Processing, IEEE Transactions on, 54(): , November [0] Julien Mairal, Francis Bach, Jean Ponce, and Guillermo Sapiro. Online learning for matrix factorization and sparse coding. J. Mach. Learn. Res., :9 60, March 200. [] Artin Armagan, David Dunson, and Merlise Clyde. Generalized beta mixtures of gaussians. In J. Shawe-Taylor, R.S. Zemel, P. Bartlett, F.C.N. Pereira, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems 24, pages [2] B. Efron. Microarrays, Empirical Bayes and the Two-Groups Model. Statistical Science, 23: 47, [3] Niall Hurley and Scott Rickard. Comparing measures of sparsity. Machine Learning for Signal Processing, MLSP IEEE Workshop on, pages 55 60, Oct [4] Yoo-Ah Kim, Stefan Wuchty, and Teresa Przytycka. Identifying causal genes and dysregulated pathways in complex diseases. PLoS computational biology, 7(3):e00095, 20. [5] Yanqing Chen, Jun Zhu, Pek Lum, Xia Yang, Shirly Pinto, Douglas MacNeil, Chunsheng Zhang, John Lamb, Stephen Edwards, Solveig Sieberts, Amy Leonardson, Lawrence Castellini, Susanna Wang, Marie-France Champy, Bin Zhang, Valur Emilsson, Sudheer Doss, Anatole Ghazalpour, Steve Horvath, Thomas Drake, Aldons Lusis, and Eric Schadt. Variations in dna elucidate molecular networks that cause disease. Nature, 452(786): ,