High Dimensional Covariance and Precision Matrix Estimation
|
|
- Jennifer Charity Blair
- 5 years ago
- Views:
Transcription
1 High Dimensional Covariance and Precision Matrix Estimation Wei Wang Washington University in St. Louis Thursday 23 rd February, 2017 Wei Wang (Washington University in St. Louis) High Dimensional Covariance and Precision Matrix Estimation Thursday 23 rd February, / 18
2 Outline 1 Introduction and Notation 2 Part I Covariance Matrix Estimation Shrinkage Estimation Sparse Estimation Factor Model-based Estimation 3 Part II Precision Matrix Estimation CLIME CONDREG Wei Wang (Washington University in St. Louis) High Dimensional Covariance and Precision Matrix Estimation Thursday 23 rd February, / 18
3 Introduction and Notation Introduction Covariance matrix marginal correlations between variables Precision (inverse covariance) matrix conditional correlations between pairs of variables given the remaining variables The estimation of covariance and precision matrices is fundamental in multivariate analysis. In high dimensional settings, sample covariance matrix has undesirable properties. p > n = singular overspreading eigenvalues under the large p small n scenario Wei Wang (Washington University in St. Louis) High Dimensional Covariance and Precision Matrix Estimation Thursday 23 rd February, / 18
4 Introduction and Notation Eigenvalues for the sample covariance matrix under the large p small n scenario Figure: Average of the largest and smallest eigenvalues of the sample covariance matrices of i.i.d samples from N(0, I) out of 100 replications where p ranges from 5 to 100 and n = 50. Wei Wang (Washington University in St. Louis) High Dimensional Covariance and Precision Matrix Estimation Thursday 23 rd February, / 18
5 Introduction and Notation Notation X i = (X i1,..., X ip ) T, i = 1,..., n, are i.i.d. samples of a p-variate random vector X = (X 1,..., X p ) T R p with Cov(X) = Σ and precision matrix Ω = Σ 1. Σ = (σ ij ) p p and Ω = (σ ij ) p p. Sample covariance matrix S n = (s jk ) p p = 1 n ( n 1 i=1 Xi X ) ( X i X ) T, where X = 1 n n i=1 X i. Operator norm of a square matrix A = (a ij ) p p : A op = λ max (A). Frobenius norm : A F = i l 1 -norm A 1 = i A = max 1 i p,1 j p a ij j a ij 2. j a ij, A 1,off = i j i a ij. Wei Wang (Washington University in St. Louis) High Dimensional Covariance and Precision Matrix Estimation Thursday 23 rd February, / 18
6 Part I Covariance Matrix Estimation Part I Covariance Matrix Estimation Wei Wang (Washington University in St. Louis) High Dimensional Covariance and Precision Matrix Estimation Thursday 23 rd February, / 18
7 Part I Covariance Matrix Estimation Shrinkage Estimation Shrinkage Estimation Ledoit and Wolf (2003) proposed the shrinkage estimation: S = λt + (1 λ)s n, where T is the target matrix and λ [0, 1] is the shrinkage parameter. T is often chosen to be positive definite and well conditioned. There are two popular target matrices: Identity matrix I diag(s 11,..., s pp) Warton (2008): the sample correlation matrix R n is regularized as ˆR(λ) = λr n + (1 λ)i, where R n = S 1/2 d S n S 1/2 d, and S d = diag(s 11,..., s pp ). Wei Wang (Washington University in St. Louis) High Dimensional Covariance and Precision Matrix Estimation Thursday 23 rd February, / 18
8 Part I Covariance Matrix Estimation Sparse Estimation Sparse Estimation: Banding, Tapering and Thresholding Banding and tapering require a natural ordering among the variables and assume that variables farther apart in the ordering are less correlated. 1. Banding Bickel and Levina (2008a) gives the k-banded estimator of Σ : B k (S n ) = [s ij 1( i j k)] p p. Here, k (0 k p) is the banding parameter which is usually chosen by a cross-validation method. Figure: Banding of a matrix whose (i, j)th entry is 0.8 i j. k = 5. Wei Wang (Washington University in St. Louis) High Dimensional Covariance and Precision Matrix Estimation Thursday 23 rd February, / 18
9 Part I Covariance Matrix Estimation Sparse Estimation 1. Banding (cont d) The banded estimator is consistent in the operator (spectral) norm, uniformly over the class of approximately bandable matrices U (α, ε) ={Σ : 0 < ε λ min(σ) λ max(σ) ε 1, max { σ j ij ; i j > k} Ck α }. Under the conditions log p 1 0 as p, n. n 2 C > 0, ε > 0 is fixed and independent of p. i α > 0 controls the rate of decay of the covariance entries σ ij as one moves away from the main diagonal. B k (S n) is not necessarily positive definite. Wei Wang (Washington University in St. Louis) High Dimensional Covariance and Precision Matrix Estimation Thursday 23 rd February, / 18
10 Part I Covariance Matrix Estimation Sparse Estimation 2. Tapering A tapered estimator of Σ with a tapering matrix W = (w ij ) p p is given by S W = S n W = (s ij w ij ) p p A smoother positive-definite tapering matrix with off diagonal entries gradually decaying to zero will ensure the positive-definiteness as well as optimal rate of convergence of the tapered estimator, eg. Cai et al.(2010) used the trapezoidal weight matrix given by 1, if i j k h, w ij = 2 i j k h, if k h < i j < k, 0, otherwise. Under the autoregressive model scenario, usually use k h = k/2. Banding is a special case of tapering with { 1, if i j k, w ij = 0, otherwise. Consistency under both the operator and Frobenius norms holds in a larger class of covariance matrices than banding where their smallest eigenvalue is allowed to be 0. Wei Wang (Washington University in St. Louis) High Dimensional Covariance and Precision Matrix Estimation Thursday 23 rd February, / 18
11 Part I Covariance Matrix Estimation Sparse Estimation Comparison of Banding and Tapering Figure: Banding and tapering of a matrix whose (i, j)th entry is 0.8 i j. Upper: banded (k = 5). Lower: tapered (k = 10, k h = 5). Wei Wang (Washington University in St. Louis) High Dimensional Covariance and Precision Matrix Estimation Thursday 23 rd February, / 18
12 Part I Covariance Matrix Estimation Sparse Estimation 3. Thresholding It does not require the variables to be ordered so that the estimator is invariant to permutation of the variables. Sparsity e.g. soft-thresholding. A soft-thresholded covariance matrix estimator is defined by applying the soft thresholding operator to S n elementwise, ˆΣ λ = S(S n, λ), where S(, λ) = sign( )( λ) + is the soft thresholding operator. The soft thresholded estimator is the solution of the following optimization problem ˆΣ λ = argmin{ 1 Σ 2 Σ Sn 2 F + λ Σ 1} = argmin Σ p i=1 j=1 p { 1 2 (σij sij)2 + λ σ ij } Wei Wang (Washington University in St. Louis) High Dimensional Covariance and Precision Matrix Estimation Thursday 23 rd February, / 18
13 Part I Covariance Matrix Estimation Sparse Estimation 3. Thresholding (cont d) Regularize the eigenvalues of S n e.g. Liu (2014): Estimation of Covariance matrices with Eigenvalue Constraints (EC2) The EC2 estimator of the correlation matrix is defined as ˆR EC2 = argmin Σ 1 2 S n Σ 2 F + λ Σ 1,off s.t. τ λ min (Σ), σ jj = 1, where τ > 0 is a desired minimum eigenvalue lower bound of the estimator. The EC2 estimator of the covariance matrix is defined as ˆΣ EC2 = S 1/2 d where S 1/2 d = diag( s 11,..., s pp ). ˆR EC2 S 1/2 d, Wei Wang (Washington University in St. Louis) High Dimensional Covariance and Precision Matrix Estimation Thursday 23 rd February, / 18
14 Part I Covariance Matrix Estimation Factor Model-based Estimation Factor Model-based Estimation In many applications, the more desirable assumption is conditional sparse, i.e. conditional on the common factors, the covariance matrix of the remaining components is sparse. Fan(2013) proposed an estimator of Σ, the principal orthogonal complement thresholding estimator (POET), which can be written as a sum of low rank and sparse matrices. Start with the spectral decomposition of the sample covariance matrix of the data, q S n = ˆλ i ê i ê T i + ˆR i=1 where q is the number of selected PCs and ˆR = (r ij ) is the matrix of residuals. The estimator is obtained by adaptively thresholding the residual matrix after taking out the first q PCs. Finding q using data-based methods is an important familiar and well-studied topic in the literature of PCA and factor analysis. Wei Wang (Washington University in St. Louis) High Dimensional Covariance and Precision Matrix Estimation Thursday 23 rd February, / 18
15 Part II Precision Matrix Estimation Part II Precision Matrix Estimation Wei Wang (Washington University in St. Louis) High Dimensional Covariance and Precision Matrix Estimation Thursday 23 rd February, / 18
16 Part II Precision Matrix Estimation CLIME Constrained l 1 -minimization for Inverse Matrix Estimation (CLIME) Cai (2011) The CLIME estimator is the solution of the following optimization problem: where λ > 0 is the tuning parameter. The solution is usually not symmetric. min Ω Ω 1 s.t. S n Ω I λ, Suppose ˆΩ 1 = (ŵ 1 ij ) p p is the solution of the above optimization problem. The final CLIME estimator is defined as ˆΩ = (ŵ ij ), where ŵ ij = ŵ ji = min{ŵ 1 ij, ŵ 1 ji}, which is demonstrated to be positive definite with high probability. Wei Wang (Washington University in St. Louis) High Dimensional Covariance and Precision Matrix Estimation Thursday 23 rd February, / 18
17 Part II Precision Matrix Estimation CONDREG CONDition number REGularized estimation (CONDREG) Won (2013) The CONDREG estimator is the solution of the following optimization problem: min tr(ωs n) log det Ω Ω where k > 0 is the tuning parameter. s.t. λ max (Ω)/λ min (Ω) k, Wei Wang (Washington University in St. Louis) High Dimensional Covariance and Precision Matrix Estimation Thursday 23 rd February, / 18
18 Q&A Thank you! Wei Wang (Washington University in St. Louis) High Dimensional Covariance and Precision Matrix Estimation Thursday 23 rd February, / 18
Permutation-invariant regularization of large covariance matrices. Liza Levina
Liza Levina Permutation-invariant covariance regularization 1/42 Permutation-invariant regularization of large covariance matrices Liza Levina Department of Statistics University of Michigan Joint work
More informationProperties of optimizations used in penalized Gaussian likelihood inverse covariance matrix estimation
Properties of optimizations used in penalized Gaussian likelihood inverse covariance matrix estimation Adam J. Rothman School of Statistics University of Minnesota October 8, 2014, joint work with Liliana
More informationSparse Permutation Invariant Covariance Estimation: Motivation, Background and Key Results
Sparse Permutation Invariant Covariance Estimation: Motivation, Background and Key Results David Prince Biostat 572 dprince3@uw.edu April 19, 2012 David Prince (UW) SPICE April 19, 2012 1 / 11 Electronic
More informationSparse Permutation Invariant Covariance Estimation: Final Talk
Sparse Permutation Invariant Covariance Estimation: Final Talk David Prince Biostat 572 dprince3@uw.edu May 31, 2012 David Prince (UW) SPICE May 31, 2012 1 / 19 Electronic Journal of Statistics Vol. 2
More informationAn efficient ADMM algorithm for high dimensional precision matrix estimation via penalized quadratic loss
An efficient ADMM algorithm for high dimensional precision matrix estimation via penalized quadratic loss arxiv:1811.04545v1 [stat.co] 12 Nov 2018 Cheng Wang School of Mathematical Sciences, Shanghai Jiao
More informationRegularized Estimation of High Dimensional Covariance Matrices. Peter Bickel. January, 2008
Regularized Estimation of High Dimensional Covariance Matrices Peter Bickel Cambridge January, 2008 With Thanks to E. Levina (Joint collaboration, slides) I. M. Johnstone (Slides) Choongsoon Bae (Slides)
More informationTuning Parameter Selection in Regularized Estimations of Large Covariance Matrices
Tuning Parameter Selection in Regularized Estimations of Large Covariance Matrices arxiv:1308.3416v1 [stat.me] 15 Aug 2013 Yixin Fang 1, Binhuan Wang 1, and Yang Feng 2 1 New York University and 2 Columbia
More informationEstimation of large dimensional sparse covariance matrices
Estimation of large dimensional sparse covariance matrices Department of Statistics UC, Berkeley May 5, 2009 Sample covariance matrix and its eigenvalues Data: n p matrix X n (independent identically distributed)
More informationBAGUS: Bayesian Regularization for Graphical Models with Unequal Shrinkage
BAGUS: Bayesian Regularization for Graphical Models with Unequal Shrinkage Lingrui Gan, Naveen N. Narisetty, Feng Liang Department of Statistics University of Illinois at Urbana-Champaign Problem Statement
More informationMultivariate Statistical Analysis
Multivariate Statistical Analysis Fall 2011 C. L. Williams, Ph.D. Lecture 4 for Applied Multivariate Analysis Outline 1 Eigen values and eigen vectors Characteristic equation Some properties of eigendecompositions
More informationHigh-dimensional covariance estimation based on Gaussian graphical models
High-dimensional covariance estimation based on Gaussian graphical models Shuheng Zhou Department of Statistics, The University of Michigan, Ann Arbor IMA workshop on High Dimensional Phenomena Sept. 26,
More informationA Multiple Testing Approach to the Regularisation of Large Sample Correlation Matrices
A Multiple Testing Approach to the Regularisation of Large Sample Correlation Matrices Natalia Bailey 1 M. Hashem Pesaran 2 L. Vanessa Smith 3 1 Department of Econometrics & Business Statistics, Monash
More informationEstimating Structured High-Dimensional Covariance and Precision Matrices: Optimal Rates and Adaptive Estimation
Estimating Structured High-Dimensional Covariance and Precision Matrices: Optimal Rates and Adaptive Estimation T. Tony Cai 1, Zhao Ren 2 and Harrison H. Zhou 3 University of Pennsylvania, University of
More informationComputationally efficient banding of large covariance matrices for ordered data and connections to banding the inverse Cholesky factor
Computationally efficient banding of large covariance matrices for ordered data and connections to banding the inverse Cholesky factor Y. Wang M. J. Daniels wang.yanpin@scrippshealth.org mjdaniels@austin.utexas.edu
More information2. LINEAR ALGEBRA. 1. Definitions. 2. Linear least squares problem. 3. QR factorization. 4. Singular value decomposition (SVD) 5.
2. LINEAR ALGEBRA Outline 1. Definitions 2. Linear least squares problem 3. QR factorization 4. Singular value decomposition (SVD) 5. Pseudo-inverse 6. Eigenvalue decomposition (EVD) 1 Definitions Vector
More informationDimension Reduction in Abundant High Dimensional Regressions
Dimension Reduction in Abundant High Dimensional Regressions Dennis Cook University of Minnesota 8th Purdue Symposium June 2012 In collaboration with Liliana Forzani & Adam Rothman, Annals of Statistics,
More informationHigh-dimensional Covariance Estimation Based On Gaussian Graphical Models
High-dimensional Covariance Estimation Based On Gaussian Graphical Models Shuheng Zhou, Philipp Rutimann, Min Xu and Peter Buhlmann February 3, 2012 Problem definition Want to estimate the covariance matrix
More informationMinimax Rate-Optimal Estimation of High- Dimensional Covariance Matrices with Incomplete Data
University of Pennsylvania ScholarlyCommons Statistics Papers Wharton Faculty Research 9-2016 Minimax Rate-Optimal Estimation of High- Dimensional Covariance Matrices with Incomplete Data T. Tony Cai University
More informationarxiv: v2 [math.st] 2 Jul 2017
A Relaxed Approach to Estimating Large Portfolios Mehmet Caner Esra Ulasan Laurent Callot A.Özlem Önder July 4, 2017 arxiv:1611.07347v2 [math.st] 2 Jul 2017 Abstract This paper considers three aspects
More information(Part 1) High-dimensional statistics May / 41
Theory for the Lasso Recall the linear model Y i = p j=1 β j X (j) i + ɛ i, i = 1,..., n, or, in matrix notation, Y = Xβ + ɛ, To simplify, we assume that the design X is fixed, and that ɛ is N (0, σ 2
More informationSparse estimation of high-dimensional covariance matrices
Sparse estimation of high-dimensional covariance matrices by Adam J. Rothman A dissertation submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy (Statistics) in The
More informationEstimation of the Global Minimum Variance Portfolio in High Dimensions
Estimation of the Global Minimum Variance Portfolio in High Dimensions Taras Bodnar, Nestor Parolya and Wolfgang Schmid 07.FEBRUARY 2014 1 / 25 Outline Introduction Random Matrix Theory: Preliminary Results
More informationMaximum Likelihood Estimation for Factor Analysis. Yuan Liao
Maximum Likelihood Estimation for Factor Analysis Yuan Liao University of Maryland Joint worth Jushan Bai June 15, 2013 High Dim. Factor Model y it = λ i f t + u it i N,t f t : common factor λ i : loading
More informationarxiv: v1 [math.st] 31 Jan 2008
Electronic Journal of Statistics ISSN: 1935-7524 Sparse Permutation Invariant arxiv:0801.4837v1 [math.st] 31 Jan 2008 Covariance Estimation Adam Rothman University of Michigan Ann Arbor, MI 48109-1107.
More informationHomework 1. Yuan Yao. September 18, 2011
Homework 1 Yuan Yao September 18, 2011 1. Singular Value Decomposition: The goal of this exercise is to refresh your memory about the singular value decomposition and matrix norms. A good reference to
More informationRobust and sparse Gaussian graphical modelling under cell-wise contamination
Robust and sparse Gaussian graphical modelling under cell-wise contamination Shota Katayama 1, Hironori Fujisawa 2 and Mathias Drton 3 1 Tokyo Institute of Technology, Japan 2 The Institute of Statistical
More information2.3. Clustering or vector quantization 57
Multivariate Statistics non-negative matrix factorisation and sparse dictionary learning The PCA decomposition is by construction optimal solution to argmin A R n q,h R q p X AH 2 2 under constraint :
More informationSparse Permutation Invariant Covariance Estimation
Sparse Permutation Invariant Covariance Estimation Adam J. Rothman University of Michigan, Ann Arbor, USA. Peter J. Bickel University of California, Berkeley, USA. Elizaveta Levina University of Michigan,
More informationNonparametric Eigenvalue-Regularized Precision or Covariance Matrix Estimator
Nonparametric Eigenvalue-Regularized Precision or Covariance Matrix Estimator Clifford Lam Department of Statistics, London School of Economics and Political Science Abstract We introduce nonparametric
More informationLecture 5 : Projections
Lecture 5 : Projections EE227C. Lecturer: Professor Martin Wainwright. Scribe: Alvin Wan Up until now, we have seen convergence rates of unconstrained gradient descent. Now, we consider a constrained minimization
More informationDimension Reduction Techniques. Presented by Jie (Jerry) Yu
Dimension Reduction Techniques Presented by Jie (Jerry) Yu Outline Problem Modeling Review of PCA and MDS Isomap Local Linear Embedding (LLE) Charting Background Advances in data collection and storage
More informationhttps://goo.gl/kfxweg KYOTO UNIVERSITY Statistical Machine Learning Theory Sparsity Hisashi Kashima kashima@i.kyoto-u.ac.jp DEPARTMENT OF INTELLIGENCE SCIENCE AND TECHNOLOGY 1 KYOTO UNIVERSITY Topics:
More informationThe Eigenvalue Problem: Perturbation Theory
Jim Lambers MAT 610 Summer Session 2009-10 Lecture 13 Notes These notes correspond to Sections 7.2 and 8.1 in the text. The Eigenvalue Problem: Perturbation Theory The Unsymmetric Eigenvalue Problem Just
More information14 Singular Value Decomposition
14 Singular Value Decomposition For any high-dimensional data analysis, one s first thought should often be: can I use an SVD? The singular value decomposition is an invaluable analysis tool for dealing
More informationPrincipal Component Analysis-I Geog 210C Introduction to Spatial Data Analysis. Chris Funk. Lecture 17
Principal Component Analysis-I Geog 210C Introduction to Spatial Data Analysis Chris Funk Lecture 17 Outline Filters and Rotations Generating co-varying random fields Translating co-varying fields into
More informationChapter 3 Transformations
Chapter 3 Transformations An Introduction to Optimization Spring, 2014 Wei-Ta Chu 1 Linear Transformations A function is called a linear transformation if 1. for every and 2. for every If we fix the bases
More informationEstimating Covariance Structure in High Dimensions
Estimating Covariance Structure in High Dimensions Ashwini Maurya Michigan State University East Lansing, MI, USA Thesis Director: Dr. Hira L. Koul Committee Members: Dr. Yuehua Cui, Dr. Hyokyoung Hong,
More informationDef. The euclidian distance between two points x = (x 1,...,x p ) t and y = (y 1,...,y p ) t in the p-dimensional space R p is defined as
MAHALANOBIS DISTANCE Def. The euclidian distance between two points x = (x 1,...,x p ) t and y = (y 1,...,y p ) t in the p-dimensional space R p is defined as d E (x, y) = (x 1 y 1 ) 2 + +(x p y p ) 2
More informationarxiv: v1 [math.st] 13 Feb 2012
Sparse Matrix Inversion with Scaled Lasso Tingni Sun and Cun-Hui Zhang Rutgers University arxiv:1202.2723v1 [math.st] 13 Feb 2012 Address: Department of Statistics and Biostatistics, Hill Center, Busch
More informationBasic Concepts in Matrix Algebra
Basic Concepts in Matrix Algebra An column array of p elements is called a vector of dimension p and is written as x p 1 = x 1 x 2. x p. The transpose of the column vector x p 1 is row vector x = [x 1
More informationHigh Dimensional Low Rank and Sparse Covariance Matrix Estimation via Convex Minimization
High Dimensional Low Rank and Sparse Covariance Matrix Estimation via Convex Minimization arxiv:1111.1133v1 [stat.me] 4 Nov 2011 Xi Luo Brown University November 10, 2018 Abstract This paper introduces
More informationSingular Value Decomposition and Principal Component Analysis (PCA) I
Singular Value Decomposition and Principal Component Analysis (PCA) I Prof Ned Wingreen MOL 40/50 Microarray review Data per array: 0000 genes, I (green) i,i (red) i 000 000+ data points! The expression
More informationLog Covariance Matrix Estimation
Log Covariance Matrix Estimation Xinwei Deng Department of Statistics University of Wisconsin-Madison Joint work with Kam-Wah Tsui (Univ. of Wisconsin-Madsion) 1 Outline Background and Motivation The Proposed
More informationSparse Permutation Invariant Covariance Estimation
Sparse Permutation Invariant Covariance Estimation Adam J. Rothman University of Michigan Ann Arbor, MI 48109-1107 e-mail: ajrothma@umich.edu Peter J. Bickel University of California Berkeley, CA 94720-3860
More informationDISCUSSION OF INFLUENTIAL FEATURE PCA FOR HIGH DIMENSIONAL CLUSTERING. By T. Tony Cai and Linjun Zhang University of Pennsylvania
Submitted to the Annals of Statistics DISCUSSION OF INFLUENTIAL FEATURE PCA FOR HIGH DIMENSIONAL CLUSTERING By T. Tony Cai and Linjun Zhang University of Pennsylvania We would like to congratulate the
More informationDimensionality Reduction: PCA. Nicholas Ruozzi University of Texas at Dallas
Dimensionality Reduction: PCA Nicholas Ruozzi University of Texas at Dallas Eigenvalues λ is an eigenvalue of a matrix A R n n if the linear system Ax = λx has at least one non-zero solution If Ax = λx
More informationAdaptive estimation of the copula correlation matrix for semiparametric elliptical copulas
Adaptive estimation of the copula correlation matrix for semiparametric elliptical copulas Department of Mathematics Department of Statistical Science Cornell University London, January 7, 2016 Joint work
More informationSparse Covariance Matrix Estimation with Eigenvalue Constraints
Sparse Covariance Matrix Estimation with Eigenvalue Constraints Han Liu and Lie Wang 2 and Tuo Zhao 3 Department of Operations Research and Financial Engineering, Princeton University 2 Department of Mathematics,
More informationEstimation of Graphical Models with Shape Restriction
Estimation of Graphical Models with Shape Restriction BY KHAI X. CHIONG USC Dornsife INE, Department of Economics, University of Southern California, Los Angeles, California 989, U.S.A. kchiong@usc.edu
More informationStructure in Data. A major objective in data analysis is to identify interesting features or structure in the data.
Structure in Data A major objective in data analysis is to identify interesting features or structure in the data. The graphical methods are very useful in discovering structure. There are basically two
More informationPosterior convergence rates for estimating large precision. matrices using graphical models
Biometrika (2013), xx, x, pp. 1 27 C 2007 Biometrika Trust Printed in Great Britain Posterior convergence rates for estimating large precision matrices using graphical models BY SAYANTAN BANERJEE Department
More informationMLCC 2015 Dimensionality Reduction and PCA
MLCC 2015 Dimensionality Reduction and PCA Lorenzo Rosasco UNIGE-MIT-IIT June 25, 2015 Outline PCA & Reconstruction PCA and Maximum Variance PCA and Associated Eigenproblem Beyond the First Principal Component
More informationProximal Gradient Descent and Acceleration. Ryan Tibshirani Convex Optimization /36-725
Proximal Gradient Descent and Acceleration Ryan Tibshirani Convex Optimization 10-725/36-725 Last time: subgradient method Consider the problem min f(x) with f convex, and dom(f) = R n. Subgradient method:
More informationDEN: Linear algebra numerical view (GEM: Gauss elimination method for reducing a full rank matrix to upper-triangular
form) Given: matrix C = (c i,j ) n,m i,j=1 ODE and num math: Linear algebra (N) [lectures] c phabala 2016 DEN: Linear algebra numerical view (GEM: Gauss elimination method for reducing a full rank matrix
More informationNext is material on matrix rank. Please see the handout
B90.330 / C.005 NOTES for Wednesday 0.APR.7 Suppose that the model is β + ε, but ε does not have the desired variance matrix. Say that ε is normal, but Var(ε) σ W. The form of W is W w 0 0 0 0 0 0 w 0
More informationDiscussion of High-dimensional autocovariance matrices and optimal linear prediction,
Electronic Journal of Statistics Vol. 9 (2015) 1 10 ISSN: 1935-7524 DOI: 10.1214/15-EJS1007 Discussion of High-dimensional autocovariance matrices and optimal linear prediction, Xiaohui Chen University
More informationx. Figure 1: Examples of univariate Gaussian pdfs N (x; µ, σ 2 ).
.8.6 µ =, σ = 1 µ = 1, σ = 1 / µ =, σ =.. 3 1 1 3 x Figure 1: Examples of univariate Gaussian pdfs N (x; µ, σ ). The Gaussian distribution Probably the most-important distribution in all of statistics
More informationarxiv: v2 [math.st] 7 Aug 2014
Sparse and Low-Rank Covariance Matrices Estimation Shenglong Zhou, Naihua Xiu, Ziyan Luo +, Lingchen Kong Department of Applied Mathematics + State Key Laboratory of Rail Traffic Control and Safety arxiv:407.4596v2
More informationStatistics for Applications. Chapter 9: Principal Component Analysis (PCA) 1/16
Statistics for Applications Chapter 9: Principal Component Analysis (PCA) 1/16 Multivariate statistics and review of linear algebra (1) Let X be a d-dimensional random vector and X 1,..., X n be n independent
More informationIndependent component analysis for functional data
Independent component analysis for functional data Hannu Oja Department of Mathematics and Statistics University of Turku Version 12.8.216 August 216 Oja (UTU) FICA Date bottom 1 / 38 Outline 1 Probability
More informationUnsupervised Learning: Dimensionality Reduction
Unsupervised Learning: Dimensionality Reduction CMPSCI 689 Fall 2015 Sridhar Mahadevan Lecture 3 Outline In this lecture, we set about to solve the problem posed in the previous lecture Given a dataset,
More informationVast Volatility Matrix Estimation for High Frequency Data
Vast Volatility Matrix Estimation for High Frequency Data Yazhen Wang National Science Foundation Yale Workshop, May 14-17, 2009 Disclaimer: My opinion, not the views of NSF Y. Wang (at NSF) 1 / 36 Outline
More informationEigenvalues and diagonalization
Eigenvalues and diagonalization Patrick Breheny November 15 Patrick Breheny BST 764: Applied Statistical Modeling 1/20 Introduction The next topic in our course, principal components analysis, revolves
More informationSparse PCA in High Dimensions
Sparse PCA in High Dimensions Jing Lei, Department of Statistics, Carnegie Mellon Workshop on Big Data and Differential Privacy Simons Institute, Dec, 2013 (Based on joint work with V. Q. Vu, J. Cho, and
More informationShrinkage Tuning Parameter Selection in Precision Matrices Estimation
arxiv:0909.1123v1 [stat.me] 7 Sep 2009 Shrinkage Tuning Parameter Selection in Precision Matrices Estimation Heng Lian Division of Mathematical Sciences School of Physical and Mathematical Sciences Nanyang
More information. a m1 a mn. a 1 a 2 a = a n
Biostat 140655, 2008: Matrix Algebra Review 1 Definition: An m n matrix, A m n, is a rectangular array of real numbers with m rows and n columns Element in the i th row and the j th column is denoted by
More informationVariable Selection for Highly Correlated Predictors
Variable Selection for Highly Correlated Predictors Fei Xue and Annie Qu arxiv:1709.04840v1 [stat.me] 14 Sep 2017 Abstract Penalty-based variable selection methods are powerful in selecting relevant covariates
More informationMatrix Rank Minimization with Applications
Matrix Rank Minimization with Applications Maryam Fazel Haitham Hindi Stephen Boyd Information Systems Lab Electrical Engineering Department Stanford University 8/2001 ACC 01 Outline Rank Minimization
More informationProbabilistic Low-Rank Matrix Completion with Adaptive Spectral Regularization Algorithms
Probabilistic Low-Rank Matrix Completion with Adaptive Spectral Regularization Algorithms François Caron Department of Statistics, Oxford STATLEARN 2014, Paris April 7, 2014 Joint work with Adrien Todeschini,
More information1 Feature Vectors and Time Series
PCA, SVD, LSI, and Kernel PCA 1 Feature Vectors and Time Series We now consider a sample x 1,..., x of objects (not necessarily vectors) and a feature map Φ such that for any object x we have that Φ(x)
More informationLecture 3: Review of Linear Algebra
ECE 83 Fall 2 Statistical Signal Processing instructor: R Nowak Lecture 3: Review of Linear Algebra Very often in this course we will represent signals as vectors and operators (eg, filters, transforms,
More informationBare minimum on matrix algebra. Psychology 588: Covariance structure and factor models
Bare minimum on matrix algebra Psychology 588: Covariance structure and factor models Matrix multiplication 2 Consider three notations for linear combinations y11 y1 m x11 x 1p b11 b 1m y y x x b b n1
More informationMethods for sparse analysis of high-dimensional data, II
Methods for sparse analysis of high-dimensional data, II Rachel Ward May 26, 2011 High dimensional data with low-dimensional structure 300 by 300 pixel images = 90, 000 dimensions 2 / 55 High dimensional
More informationLecture 3: Review of Linear Algebra
ECE 83 Fall 2 Statistical Signal Processing instructor: R Nowak, scribe: R Nowak Lecture 3: Review of Linear Algebra Very often in this course we will represent signals as vectors and operators (eg, filters,
More informationComputational and Statistical Aspects of Statistical Machine Learning. John Lafferty Department of Statistics Retreat Gleacher Center
Computational and Statistical Aspects of Statistical Machine Learning John Lafferty Department of Statistics Retreat Gleacher Center Outline Modern nonparametric inference for high dimensional data Nonparametric
More informationBootstrapping factor models with cross sectional dependence
Bootstrapping factor models with cross sectional dependence Sílvia Gonçalves and Benoit Perron University of Western Ontario and Université de Montréal, CIREQ, CIRAO ovember, 06 Abstract We consider bootstrap
More informationMethods for sparse analysis of high-dimensional data, II
Methods for sparse analysis of high-dimensional data, II Rachel Ward May 23, 2011 High dimensional data with low-dimensional structure 300 by 300 pixel images = 90, 000 dimensions 2 / 47 High dimensional
More informationNon white sample covariance matrices.
Non white sample covariance matrices. S. Péché, Université Grenoble 1, joint work with O. Ledoit, Uni. Zurich 17-21/05/2010, Université Marne la Vallée Workshop Probability and Geometry in High Dimensions
More informationPrinciple Components Analysis (PCA) Relationship Between a Linear Combination of Variables and Axes Rotation for PCA
Principle Components Analysis (PCA) Relationship Between a Linear Combination of Variables and Axes Rotation for PCA Principle Components Analysis: Uses one group of variables (we will call this X) In
More informationApproximate Principal Components Analysis of Large Data Sets
Approximate Principal Components Analysis of Large Data Sets Daniel J. McDonald Department of Statistics Indiana University mypage.iu.edu/ dajmcdon April 27, 2016 Approximation-Regularization for Analysis
More informationUNIT 6: The singular value decomposition.
UNIT 6: The singular value decomposition. María Barbero Liñán Universidad Carlos III de Madrid Bachelor in Statistics and Business Mathematical methods II 2011-2012 A square matrix is symmetric if A T
More informationOn Spectral Factorization and Riccati Equations for Time-Varying Systems in Discrete Time
On Spectral Factorization and Riccati Equations for Time-Varying Systems in Discrete Time Alle-Jan van der Veen and Michel Verhaegen Delft University of Technology Department of Electrical Engineering
More informationLinear Dimensionality Reduction
Outline Hong Chang Institute of Computing Technology, Chinese Academy of Sciences Machine Learning Methods (Fall 2012) Outline Outline I 1 Introduction 2 Principal Component Analysis 3 Factor Analysis
More informationThe University of Texas at Austin Department of Electrical and Computer Engineering. EE381V: Large Scale Learning Spring 2013.
The University of Texas at Austin Department of Electrical and Computer Engineering EE381V: Large Scale Learning Spring 2013 Assignment Two Caramanis/Sanghavi Due: Tuesday, Feb. 19, 2013. Computational
More informationJournal of Multivariate Analysis. Consistency of sparse PCA in High Dimension, Low Sample Size contexts
Journal of Multivariate Analysis 5 (03) 37 333 Contents lists available at SciVerse ScienceDirect Journal of Multivariate Analysis journal homepage: www.elsevier.com/locate/jmva Consistency of sparse PCA
More informationOptimal spectral shrinkage and PCA with heteroscedastic noise
Optimal spectral shrinage and PCA with heteroscedastic noise William Leeb and Elad Romanov Abstract This paper studies the related problems of denoising, covariance estimation, and principal component
More informationRobust Principal Component Analysis
ELE 538B: Mathematics of High-Dimensional Data Robust Principal Component Analysis Yuxin Chen Princeton University, Fall 2018 Disentangling sparse and low-rank matrices Suppose we are given a matrix M
More informationLecture 7. Econ August 18
Lecture 7 Econ 2001 2015 August 18 Lecture 7 Outline First, the theorem of the maximum, an amazing result about continuity in optimization problems. Then, we start linear algebra, mostly looking at familiar
More informationELE 538B: Mathematics of High-Dimensional Data. Spectral methods. Yuxin Chen Princeton University, Fall 2018
ELE 538B: Mathematics of High-Dimensional Data Spectral methods Yuxin Chen Princeton University, Fall 2018 Outline A motivating application: graph clustering Distance and angles between two subspaces Eigen-space
More informationSection 3.9. Matrix Norm
3.9. Matrix Norm 1 Section 3.9. Matrix Norm Note. We define several matrix norms, some similar to vector norms and some reflecting how multiplication by a matrix affects the norm of a vector. We use matrix
More informationSTAT 309: MATHEMATICAL COMPUTATIONS I FALL 2013 PROBLEM SET 2
STAT 309: MATHEMATICAL COMPUTATIONS I FALL 2013 PROBLEM SET 2 1. You are not allowed to use the svd for this problem, i.e. no arguments should depend on the svd of A or A. Let W be a subspace of C n. The
More informationCS540 Machine learning Lecture 5
CS540 Machine learning Lecture 5 1 Last time Basis functions for linear regression Normal equations QR SVD - briefly 2 This time Geometry of least squares (again) SVD more slowly LMS Ridge regression 3
More informationSecond-Order Inference for Gaussian Random Curves
Second-Order Inference for Gaussian Random Curves With Application to DNA Minicircles Victor Panaretos David Kraus John Maddocks Ecole Polytechnique Fédérale de Lausanne Panaretos, Kraus, Maddocks (EPFL)
More informationBootstrapping factor models with cross sectional dependence
Bootstrapping factor models with cross sectional dependence Sílvia Gonçalves and Benoit Perron McGill University, CIREQ, CIRAO and Université de Montréal, CIREQ, CIRAO ovember 4, 07 Abstract We consider
More informationLECTURE NOTE #10 PROF. ALAN YUILLE
LECTURE NOTE #10 PROF. ALAN YUILLE 1. Principle Component Analysis (PCA) One way to deal with the curse of dimensionality is to project data down onto a space of low dimensions, see figure (1). Figure
More informationMATH 829: Introduction to Data Mining and Analysis Graphical Models II - Gaussian Graphical Models
1/13 MATH 829: Introduction to Data Mining and Analysis Graphical Models II - Gaussian Graphical Models Dominique Guillot Departments of Mathematical Sciences University of Delaware May 4, 2016 Recall
More informationSTAT 100C: Linear models
STAT 100C: Linear models Arash A. Amini April 27, 2018 1 / 1 Table of Contents 2 / 1 Linear Algebra Review Read 3.1 and 3.2 from text. 1. Fundamental subspace (rank-nullity, etc.) Im(X ) = ker(x T ) R
More informationUnsupervised dimensionality reduction
Unsupervised dimensionality reduction Guillaume Obozinski Ecole des Ponts - ParisTech SOCN course 2014 Guillaume Obozinski Unsupervised dimensionality reduction 1/30 Outline 1 PCA 2 Kernel PCA 3 Multidimensional
More informationApplications of Randomized Methods for Decomposing and Simulating from Large Covariance Matrices
Applications of Randomized Methods for Decomposing and Simulating from Large Covariance Matrices Vahid Dehdari and Clayton V. Deutsch Geostatistical modeling involves many variables and many locations.
More informationTuning-parameter selection in regularized estimations of large covariance matrices
Journal of Statistical Computation and Simulation, 2016 Vol. 86, No. 3, 494 509, http://dx.doi.org/10.1080/00949655.2015.1017823 Tuning-parameter selection in regularized estimations of large covariance
More information