Sparse representation classification and positive L1 minimization
|
|
|
- Barnaby McKenzie
- 5 years ago
- Views:
Transcription
1 Sparse representation classification and positive L1 minimization Cencheng Shen Joint Work with Li Chen, Carey E. Priebe Applied Mathematics and Statistics Johns Hopkins University, August 5, 2014 Cencheng Shen (JHU) JSM2014 Presentation August 5, / 30
2 Overview 1 Introduction 2 Numerical experiments 3 Conclusion Cencheng Shen (JHU) JSM2014 Presentation August 5, / 30
3 Section 1 Introduction Cencheng Shen (JHU) JSM2014 Presentation August 5, / 30
4 Sparse representation classification? Our motivation comes from the sparse representation classification (SRC) proposed in Wright et al [1]. It is a simple and intuitive classification procedure making use of L1 minimization, and argued to strike a balance between nearest-neighbor and nearest-subspace classifiers, while being more discriminative than both. Numerically shown to be a superior classifier for image data, robust against dimension reduction and data contamination. Cencheng Shen (JHU) JSM2014 Presentation August 5, / 30
5 The SRC Algorithm Set-up: An m n training matrix X, and the labels y i [1,..., K] corresponding to each column x i of X. And an m 1 testing vector x for classification. All data are normalized to column-wise unit norm. Find a sparse representation of x in terms of X : Solve ˆβ = arg min β 1 subject to x X β 2 ɛ. (1) We use homotopy by Osborne et al [2] and orthogonal matching pursuit (OMP) by Tropp 2004 [3] to solve this, and bound the number of maximal iterations without using ɛ in our work. Classify x by the sparse representation ˆβ: g(x) = arg min x X ˆβ k 2, (2) k=1,...,k where ˆβ k is the class-conditional sparse representation with ˆβ k (i) = ˆβ(i) if y i = k and ˆβ k (i) = 0 otherwise. Break ties deterministically. Cencheng Shen (JHU) JSM2014 Presentation August 5, / 30
6 Theoretical guarantee? Wright et al [1] argues that SRC works well for the image data, because empirically different classes of images lie on different subspaces. Towards the same direction, Elhamifar and Vidal 2013 [4] proves a sufficient condition for L1 minimization to only choose points from the same subspace, so that sparse representation can work optimally for spectral clustering on data from multiple subspaces. Chen et al [5] applies SRC to vertex classification using adjacency matrices and OMP, which exhibits robust performance on graph data, but not always the best classifier. But adjacency matrix does not enjoy the subspace property. Also adjacency matrix has m = n such that the residual by L1 minimization is usually high at small sparsity limit. Cencheng Shen (JHU) JSM2014 Presentation August 5, / 30
7 Our questions on SRC and L1 minimization Q1. Since many data do not have the subspace property, is SRC applicable beyond the subspace property? Q2. The key step of SRC is the L1 minimization step (also widely known as Lasso by Tibshirani 1996 [6]). Since real data is usually noisy and may be high-dimensional (like the (dis)similarity matrices which we care a lot), and a good residual cut-off is hard to estimate, is there a better way to stop the L1 minimization without explicit model selection? (e.g., Efron et al [7] uses Mallows selection criteria for Lasso, Wright et al [1] uses a simple cut-off ɛ = 0.05, Elhamifar and Vidal 2013 [4] assumes perfect recovery for their theorem.) Q3. As a greedy algorithm that is very easy to implement, OMP is very popular to give an approximate solution of the exact L1 minimization, and a suitable tool for large data processing. Is there any guarantee on its equivalence with L1 minimization? (This is discussed by both Efron et al [7] and Donoho and Tsaig 2006 [8]) Cencheng Shen (JHU) JSM2014 Presentation August 5, / 30
8 A simple guarantee on SRC performance In our working paper Shen et al [9], we provide a very coarse error bound of SRC based on within-class principal angles and between-class principal angles. In short, if the former is smaller than the latter, SRC may succeed. This can help us find meaningful models that can work with SRC beyond the subspace property. For example, we further prove that SRC is a consistent classifier for degree-corrected SBM (under one mild condition) applied on the adjacency matrix. It is conceptually similar to the condition in Elhamifar and Vidal 2013 [4], where they also impose a condition so that data on the same subspace is sufficiently close comparing to data of different subspaces. But there are intrinsic differences in the assumption, condition and the proof. Cencheng Shen (JHU) JSM2014 Presentation August 5, / 30
9 And... Q1 partly solved?! But finite-sample performance is not necessarily optimal. What about Q2 and Q3? Let us use positive L1 minimization! Cencheng Shen (JHU) JSM2014 Presentation August 5, / 30
10 Positive L1 minimization Instead of the usual L1 minimization, we add one more constraint ˆβ = arg min β 1 subject to x X β 2 ɛ and β 0 n 1, (3) where the sign is entry-wise. The positive constraint can be easily added to homotopy and OMP with no extra computation. It is briefly mentioned in the Lasso implementation using homotopy in Efron et al [7], and called positive Lasso. So far we cannot find any other investigation on positive L1 minimization, in spite of the rich literature in L1/L0 area. Cencheng Shen (JHU) JSM2014 Presentation August 5, / 30
11 Impact on SRC? It usually stops much earlier than usual L1 minimization. And we prove that OMP is more likely to be equivalent to L1 or the true model under the positive constraint. It is a bias-variance trade-off? Cencheng Shen (JHU) JSM2014 Presentation August 5, / 30
12 Section 2 Numerical experiments Cencheng Shen (JHU) JSM2014 Presentation August 5, / 30
13 Numerical experiments For all the data, we randomly split half for training and the other half for testing, and plot the hold-out SRC error against the sparsity level, with iteration limit being 100. Then we plot the sparsity level histogram of usual/positive OMP/homotopy. In order to show that OMP and L1 is more likely to be equivalent, we plot the histogram of the following matching statistic p = n I ˆβ(i)>0 I β(i)>0 / min{ I ˆβ(i)>0, I β(i)>0 }. (4) i=1 So if ˆβ and β have nonzero entries at same positions (or a subset of another), p = 1; and increasing mismatch will degrade the p towards 0. We also show the residual histogram of usual/positive L1 minimization. Cencheng Shen (JHU) JSM2014 Presentation August 5, / 30
14 SRC errors for Extended Yale B Images Extended Yale B database has 2414 face images of 38 individuals under various poses and lighting conditions. So m = 1024, n = 1207, and K = 38. SRC under positive constraint is roughly worse by Cencheng Shen (JHU) JSM2014 Presentation August 5, / 30
15 SRC errors on CMU PIE Images The CMU PIE database has images of 68 individuals under various poses, illuminations and expressions. m = 1024, n = 5777, and K = 68. SRC under positive constraint is roughly worse by less than Cencheng Shen (JHU) JSM2014 Presentation August 5, / 30
16 L1 comparison in sparsity level for Yale Image The left side is the number of selected data by usual homotopy/omp, the right side is that for positive homotopy/omp. Cencheng Shen (JHU) JSM2014 Presentation August 5, / 30
17 OMP L1 equivalence for Yale Image The left is OMP and homotopy equivalence without positive constraint, the right is with positive constraint. Cencheng Shen (JHU) JSM2014 Presentation August 5, / 30
18 Residuals for Yale Image The left is the residual of usual homotopy, the right is the residual of positive homotopy. CMU PIE dataset has similar plots too! Cencheng Shen (JHU) JSM2014 Presentation August 5, / 30
19 SRC errors on Political Blogs Network The Political Blogs data is a directed graph of 1490 blogs on conservatives and libertarians, so we have a adjacency matrix. Among which 1224 vertices have edges, so m = 1224, n = 612 and K = 2. The data can be modeled by DC-SBM. We also add LDA/9NN ASE for comparison. Cencheng Shen (JHU) JSM2014 Presentation August 5, / 30
20 L1 comparison in sparsity level for PolBlogs Network Cencheng Shen (JHU) JSM2014 Presentation August 5, / 30
21 OMP L1 equivalence for PolBlogs Network Cencheng Shen (JHU) JSM2014 Presentation August 5, / 30
22 Residuals for PolBlogs Network Cencheng Shen (JHU) JSM2014 Presentation August 5, / 30
23 SRC errors on YouTube Video This is a dataset on YouTube game videos containing videos with 31 game genres. We randomly use videos and vision hog feature, where we have m = 650, n = 5000, and K = 31. We also add LDA/9NN PCA for comparison. Cencheng Shen (JHU) JSM2014 Presentation August 5, / 30
24 L1 comparison in sparsity level for YouTube Video Cencheng Shen (JHU) JSM2014 Presentation August 5, / 30
25 OMP L1 equivalence for YouTube Video Cencheng Shen (JHU) JSM2014 Presentation August 5, / 30
26 Residuals for YouTube Video Cencheng Shen (JHU) JSM2014 Presentation August 5, / 30
27 Section 3 Conclusion Cencheng Shen (JHU) JSM2014 Presentation August 5, / 30
28 Conclusion In this talk, we find partial solutions to our three questions. Q1 We extend SRC beyond the subspace property and generalize it to the graph data theoretically. We also argue that SRC with positive constraint is reasonable. Q2 We show that positive L1 minimization terminates much earlier and yield a more parsimonious solution than usual L1 minimization (though mostly numerically). This is achieved without any additional model selection, at the cost of slightly larger residual. Q3 From an algorithmic point of view, we show that OMP is more likely to be equivalent to the exact L1 minimization/true model under the positive constraint. The improvement is very significant in all our experiments for the equivalence of OMP and homotopy. However, there are still many unknowns... Cencheng Shen (JHU) JSM2014 Presentation August 5, / 30
29 References I J. Wright, A. Y. Yang, A. Ganesh, S. Shankar, and Y. Ma, Robust face recognition via sparse representation, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 31, no. 2, pp , M. R. Osborne, B. Presnell, and B. A. Turlach, A new approach to variable selection in least squares problems, IMA Journal of Numerical Analysis, vol. 20, pp , J. Tropp, Greed is good: Algorithmic results for sparse approximation, IEEE Transactions on Information Theory, vol. 50, no. 10, pp , E. Elhamifar and R. Vidal, Sparse subspace clustering: Algorithm, theory, and applications, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35, no. 11, pp , L. Chen, J. Vogelstein, and C. E. Priebe, Robust vertex classification, submitted, on arxiv, R. Tibshirani, Regression shrinkage and selection via the lasso, Journal of the Royal Statistical Society: Series B, vol. 58, no. 1, pp , B. Efron, T. Hastie, I. Johnstone, and R. Tibshirani, Least angle regression, The Annals of Statistics, vol. 32, no. 2, pp , D. Donoho and Y. Tsaig, Fast solution of l1-norm minimization problems when the solution may be sparse, preprint, C. Shen, L. Chen, and C. E. Priebe, Sparse representation classification and positive l1 minimization, to be submitted, Cencheng Shen (JHU) JSM2014 Presentation August 5, / 30
30 Thank you! Cencheng Shen (JHU) JSM2014 Presentation August 5, / 30
Scalable Subspace Clustering
Scalable Subspace Clustering René Vidal Center for Imaging Science, Laboratory for Computational Sensing and Robotics, Institute for Computational Medicine, Department of Biomedical Engineering, Johns
Sparsity in Underdetermined Systems
Sparsity in Underdetermined Systems Department of Statistics Stanford University August 19, 2005 Classical Linear Regression Problem X n y p n 1 > Given predictors and response, y Xβ ε = + ε N( 0, σ 2
Group Sparse Non-negative Matrix Factorization for Multi-Manifold Learning
LIU, LU, GU: GROUP SPARSE NMF FOR MULTI-MANIFOLD LEARNING 1 Group Sparse Non-negative Matrix Factorization for Multi-Manifold Learning Xiangyang Liu 1,2 liuxy@sjtu.edu.cn Hongtao Lu 1 htlu@sjtu.edu.cn
Robust Principal Component Analysis
ELE 538B: Mathematics of High-Dimensional Data Robust Principal Component Analysis Yuxin Chen Princeton University, Fall 2018 Disentangling sparse and low-rank matrices Suppose we are given a matrix M
When Dictionary Learning Meets Classification
When Dictionary Learning Meets Classification Bufford, Teresa 1 Chen, Yuxin 2 Horning, Mitchell 3 Shee, Liberty 1 Mentor: Professor Yohann Tendero 1 UCLA 2 Dalhousie University 3 Harvey Mudd College August
New Coherence and RIP Analysis for Weak. Orthogonal Matching Pursuit
New Coherence and RIP Analysis for Wea 1 Orthogonal Matching Pursuit Mingrui Yang, Member, IEEE, and Fran de Hoog arxiv:1405.3354v1 [cs.it] 14 May 2014 Abstract In this paper we define a new coherence
Basis Pursuit Denoising and the Dantzig Selector
BPDN and DS p. 1/16 Basis Pursuit Denoising and the Dantzig Selector West Coast Optimization Meeting University of Washington Seattle, WA, April 28 29, 2007 Michael Friedlander and Michael Saunders Dept
... SPARROW. SPARse approximation Weighted regression. Pardis Noorzad. Department of Computer Engineering and IT Amirkabir University of Technology
..... SPARROW SPARse approximation Weighted regression Pardis Noorzad Department of Computer Engineering and IT Amirkabir University of Technology Université de Montréal March 12, 2012 SPARROW 1/47 .....
Nonnegative Matrix Factorization Clustering on Multiple Manifolds
Proceedings of the Twenty-Fourth AAAI Conference on Artificial Intelligence (AAAI-10) Nonnegative Matrix Factorization Clustering on Multiple Manifolds Bin Shen, Luo Si Department of Computer Science,
Sparse Subspace Clustering
Sparse Subspace Clustering Based on Sparse Subspace Clustering: Algorithm, Theory, and Applications by Elhamifar and Vidal (2013) Alex Gutierrez CSCI 8314 March 2, 2017 Outline 1 Motivation and Background
Orthogonal Matching Pursuit for Sparse Signal Recovery With Noise
Orthogonal Matching Pursuit for Sparse Signal Recovery With Noise The MIT Faculty has made this article openly available. Please share how this access benefits you. Your story matters. Citation As Published
SUBSPACE CLUSTERING WITH DENSE REPRESENTATIONS. Eva L. Dyer, Christoph Studer, Richard G. Baraniuk
SUBSPACE CLUSTERING WITH DENSE REPRESENTATIONS Eva L. Dyer, Christoph Studer, Richard G. Baraniuk Rice University; e-mail: {e.dyer, studer, richb@rice.edu} ABSTRACT Unions of subspaces have recently been
Sparse Linear Models (10/7/13)
STA56: Probabilistic machine learning Sparse Linear Models (0/7/) Lecturer: Barbara Engelhardt Scribes: Jiaji Huang, Xin Jiang, Albert Oh Sparsity Sparsity has been a hot topic in statistics and machine
Scalable Sparse Subspace Clustering by Orthogonal Matching Pursuit
Scalable Sparse Subspace Clustering by Orthogonal Matching Pursuit Chong You, Daniel P. Robinson, René Vidal Johns Hopkins University, Baltimore, MD, 21218, USA Abstract Subspace clustering methods based
CSC 576: Variants of Sparse Learning
CSC 576: Variants of Sparse Learning Ji Liu Department of Computer Science, University of Rochester October 27, 205 Introduction Our previous note basically suggests using l norm to enforce sparsity in
An Homotopy Algorithm for the Lasso with Online Observations
An Homotopy Algorithm for the Lasso with Online Observations Pierre J. Garrigues Department of EECS Redwood Center for Theoretical Neuroscience University of California Berkeley, CA 94720 garrigue@eecs.berkeley.edu
DISCUSSION OF INFLUENTIAL FEATURE PCA FOR HIGH DIMENSIONAL CLUSTERING. By T. Tony Cai and Linjun Zhang University of Pennsylvania
Submitted to the Annals of Statistics DISCUSSION OF INFLUENTIAL FEATURE PCA FOR HIGH DIMENSIONAL CLUSTERING By T. Tony Cai and Linjun Zhang University of Pennsylvania We would like to congratulate the
MLCC 2018 Variable Selection and Sparsity. Lorenzo Rosasco UNIGE-MIT-IIT
MLCC 2018 Variable Selection and Sparsity Lorenzo Rosasco UNIGE-MIT-IIT Outline Variable Selection Subset Selection Greedy Methods: (Orthogonal) Matching Pursuit Convex Relaxation: LASSO & Elastic Net
Analysis of Greedy Algorithms
Analysis of Greedy Algorithms Jiahui Shen Florida State University Oct.26th Outline Introduction Regularity condition Analysis on orthogonal matching pursuit Analysis on forward-backward greedy algorithm
A tutorial on sparse modeling. Outline:
A tutorial on sparse modeling. Outline: 1. Why? 2. What? 3. How. 4. no really, why? Sparse modeling is a component in many state of the art signal processing and machine learning tasks. image processing
SUBSPACE CLUSTERING WITH DENSE REPRESENTATIONS. Eva L. Dyer, Christoph Studer, Richard G. Baraniuk. ECE Department, Rice University, Houston, TX
SUBSPACE CLUSTERING WITH DENSE REPRESENTATIONS Eva L. Dyer, Christoph Studer, Richard G. Baraniuk ECE Department, Rice University, Houston, TX ABSTRACT Unions of subspaces have recently been shown to provide
ISyE 691 Data mining and analytics
ISyE 691 Data mining and analytics Regression Instructor: Prof. Kaibo Liu Department of Industrial and Systems Engineering UW-Madison Email: kliu8@wisc.edu Office: Room 3017 (Mechanical Engineering Building)
Tractable Upper Bounds on the Restricted Isometry Constant
Tractable Upper Bounds on the Restricted Isometry Constant Alex d Aspremont, Francis Bach, Laurent El Ghaoui Princeton University, École Normale Supérieure, U.C. Berkeley. Support from NSF, DHS and Google.
Rice University. Endogenous Sparse Recovery. Eva L. Dyer. A Thesis Submitted in Partial Fulfillment of the Requirements for the Degree
Rice University Endogenous Sparse Recovery by Eva L. Dyer A Thesis Submitted in Partial Fulfillment of the Requirements for the Degree Masters of Science Approved, Thesis Committee: Dr. Richard G. Baraniuk,
TECHNICAL REPORT NO. 1091r. A Note on the Lasso and Related Procedures in Model Selection
DEPARTMENT OF STATISTICS University of Wisconsin 1210 West Dayton St. Madison, WI 53706 TECHNICAL REPORT NO. 1091r April 2004, Revised December 2004 A Note on the Lasso and Related Procedures in Model
An Introduction to Sparse Approximation
An Introduction to Sparse Approximation Anna C. Gilbert Department of Mathematics University of Michigan Basic image/signal/data compression: transform coding Approximate signals sparsely Compress images,
COMS 4771 Lecture Fixed-design linear regression 2. Ridge and principal components regression 3. Sparse regression and Lasso
COMS 477 Lecture 6. Fixed-design linear regression 2. Ridge and principal components regression 3. Sparse regression and Lasso / 2 Fixed-design linear regression Fixed-design linear regression A simplified
A Modern Look at Classical Multivariate Techniques
A Modern Look at Classical Multivariate Techniques Yoonkyung Lee Department of Statistics The Ohio State University March 16-20, 2015 The 13th School of Probability and Statistics CIMAT, Guanajuato, Mexico
Bias-free Sparse Regression with Guaranteed Consistency
Bias-free Sparse Regression with Guaranteed Consistency Wotao Yin (UCLA Math) joint with: Stanley Osher, Ming Yan (UCLA) Feng Ruan, Jiechao Xiong, Yuan Yao (Peking U) UC Riverside, STATS Department March
Saharon Rosset 1 and Ji Zhu 2
Aust. N. Z. J. Stat. 46(3), 2004, 505 510 CORRECTED PROOF OF THE RESULT OF A PREDICTION ERROR PROPERTY OF THE LASSO ESTIMATOR AND ITS GENERALIZATION BY HUANG (2003) Saharon Rosset 1 and Ji Zhu 2 IBM T.J.
Elaine T. Hale, Wotao Yin, Yin Zhang
, Wotao Yin, Yin Zhang Department of Computational and Applied Mathematics Rice University McMaster University, ICCOPT II-MOPTA 2007 August 13, 2007 1 with Noise 2 3 4 1 with Noise 2 3 4 1 with Noise 2
Introduction to Machine Learning. PCA and Spectral Clustering. Introduction to Machine Learning, Slides: Eran Halperin
1 Introduction to Machine Learning PCA and Spectral Clustering Introduction to Machine Learning, 2013-14 Slides: Eran Halperin Singular Value Decomposition (SVD) The singular value decomposition (SVD)
Learning Multiple Tasks with a Sparse Matrix-Normal Penalty
Learning Multiple Tasks with a Sparse Matrix-Normal Penalty Yi Zhang and Jeff Schneider NIPS 2010 Presented by Esther Salazar Duke University March 25, 2011 E. Salazar (Reading group) March 25, 2011 1
Sparse regression. Optimization-Based Data Analysis. Carlos Fernandez-Granda
Sparse regression Optimization-Based Data Analysis http://www.cims.nyu.edu/~cfgranda/pages/obda_spring16 Carlos Fernandez-Granda 3/28/2016 Regression Least-squares regression Example: Global warming Logistic
OVERLAPPING ANIMAL SOUND CLASSIFICATION USING SPARSE REPRESENTATION
OVERLAPPING ANIMAL SOUND CLASSIFICATION USING SPARSE REPRESENTATION Na Lin, Haixin Sun Xiamen University Key Laboratory of Underwater Acoustic Communication and Marine Information Technology, Ministry
Uncorrelated Multilinear Principal Component Analysis through Successive Variance Maximization
Uncorrelated Multilinear Principal Component Analysis through Successive Variance Maximization Haiping Lu 1 K. N. Plataniotis 1 A. N. Venetsanopoulos 1,2 1 Department of Electrical & Computer Engineering,
Linear Methods for Regression. Lijun Zhang
Linear Methods for Regression Lijun Zhang zlj@nju.edu.cn http://cs.nju.edu.cn/zlj Outline Introduction Linear Regression Models and Least Squares Subset Selection Shrinkage Methods Methods Using Derived
Semi-supervised Dictionary Learning Based on Hilbert-Schmidt Independence Criterion
Semi-supervised ictionary Learning Based on Hilbert-Schmidt Independence Criterion Mehrdad J. Gangeh 1, Safaa M.A. Bedawi 2, Ali Ghodsi 3, and Fakhri Karray 2 1 epartments of Medical Biophysics, and Radiation
Noisy and Missing Data Regression: Distribution-Oblivious Support Recovery
: Distribution-Oblivious Support Recovery Yudong Chen Department of Electrical and Computer Engineering The University of Texas at Austin Austin, TX 7872 Constantine Caramanis Department of Electrical
Signal Recovery from Permuted Observations
EE381V Course Project Signal Recovery from Permuted Observations 1 Problem Shanshan Wu (sw33323) May 8th, 2015 We start with the following problem: let s R n be an unknown n-dimensional real-valued signal,
A New Combined Approach for Inference in High-Dimensional Regression Models with Correlated Variables
A New Combined Approach for Inference in High-Dimensional Regression Models with Correlated Variables Niharika Gauraha and Swapan Parui Indian Statistical Institute Abstract. We consider the problem of
Machine Learning 11. week
Machine Learning 11. week Feature Extraction-Selection Dimension reduction PCA LDA 1 Feature Extraction Any problem can be solved by machine learning methods in case of that the system must be appropriately
ECE G: Special Topics in Signal Processing: Sparsity, Structure, and Inference
ECE 18-898G: Special Topics in Signal Processing: Sparsity, Structure, and Inference Sparse Recovery using L1 minimization - algorithms Yuejie Chi Department of Electrical and Computer Engineering Spring
Statistical Machine Learning
Statistical Machine Learning Christoph Lampert Spring Semester 2015/2016 // Lecture 12 1 / 36 Unsupervised Learning Dimensionality Reduction 2 / 36 Dimensionality Reduction Given: data X = {x 1,..., x
Learning the Semantic Correlation: An Alternative Way to Gain from Unlabeled Text
Learning the Semantic Correlation: An Alternative Way to Gain from Unlabeled Text Yi Zhang Machine Learning Department Carnegie Mellon University yizhang1@cs.cmu.edu Jeff Schneider The Robotics Institute
Smoothly Clipped Absolute Deviation (SCAD) for Correlated Variables
Smoothly Clipped Absolute Deviation (SCAD) for Correlated Variables LIB-MA, FSSM Cadi Ayyad University (Morocco) COMPSTAT 2010 Paris, August 22-27, 2010 Motivations Fan and Li (2001), Zou and Li (2008)
Greedy Signal Recovery and Uniform Uncertainty Principles
Greedy Signal Recovery and Uniform Uncertainty Principles SPIE - IE 2008 Deanna Needell Joint work with Roman Vershynin UC Davis, January 2008 Greedy Signal Recovery and Uniform Uncertainty Principles
Linear Model Selection and Regularization
Linear Model Selection and Regularization Recall the linear model Y = β 0 + β 1 X 1 + + β p X p + ɛ. In the lectures that follow, we consider some approaches for extending the linear model framework. In
Graph based Subspace Segmentation. Canyi Lu National University of Singapore Nov. 21, 2013
Graph based Subspace Segmentation Canyi Lu National University of Singapore Nov. 21, 2013 Content Subspace Segmentation Problem Related Work Sparse Subspace Clustering (SSC) Low-Rank Representation (LRR)
Pathwise coordinate optimization
Stanford University 1 Pathwise coordinate optimization Jerome Friedman, Trevor Hastie, Holger Hoefling, Robert Tibshirani Stanford University Acknowledgements: Thanks to Stephen Boyd, Michael Saunders,
SOLVING NON-CONVEX LASSO TYPE PROBLEMS WITH DC PROGRAMMING. Gilles Gasso, Alain Rakotomamonjy and Stéphane Canu
SOLVING NON-CONVEX LASSO TYPE PROBLEMS WITH DC PROGRAMMING Gilles Gasso, Alain Rakotomamonjy and Stéphane Canu LITIS - EA 48 - INSA/Universite de Rouen Avenue de l Université - 768 Saint-Etienne du Rouvray
Lecture Notes 10: Matrix Factorization
Optimization-based data analysis Fall 207 Lecture Notes 0: Matrix Factorization Low-rank models. Rank- model Consider the problem of modeling a quantity y[i, j] that depends on two indices i and j. To
Sparse Approximation and Variable Selection
Sparse Approximation and Variable Selection Lorenzo Rosasco 9.520 Class 07 February 26, 2007 About this class Goal To introduce the problem of variable selection, discuss its connection to sparse approximation
Stability and Robustness of Weak Orthogonal Matching Pursuits
Stability and Robustness of Weak Orthogonal Matching Pursuits Simon Foucart, Drexel University Abstract A recent result establishing, under restricted isometry conditions, the success of sparse recovery
Sparsity Models. Tong Zhang. Rutgers University. T. Zhang (Rutgers) Sparsity Models 1 / 28
Sparsity Models Tong Zhang Rutgers University T. Zhang (Rutgers) Sparsity Models 1 / 28 Topics Standard sparse regression model algorithms: convex relaxation and greedy algorithm sparse recovery analysis:
L 2,1 Norm and its Applications
L 2, Norm and its Applications Yale Chang Introduction According to the structure of the constraints, the sparsity can be obtained from three types of regularizers for different purposes.. Flat Sparsity.
Pattern Recognition 2
Pattern Recognition 2 KNN,, Dr. Terence Sim School of Computing National University of Singapore Outline 1 2 3 4 5 Outline 1 2 3 4 5 The Bayes Classifier is theoretically optimum. That is, prob. of error
On Algorithms for Solving Least Squares Problems under an L 1 Penalty or an L 1 Constraint
On Algorithms for Solving Least Squares Problems under an L 1 Penalty or an L 1 Constraint B.A. Turlach School of Mathematics and Statistics (M19) The University of Western Australia 35 Stirling Highway,
Machine Learning for OR & FE
Machine Learning for OR & FE Regression II: Regularization and Shrinkage Methods Martin Haugh Department of Industrial Engineering and Operations Research Columbia University Email: martin.b.haugh@gmail.com
2.3. Clustering or vector quantization 57
Multivariate Statistics non-negative matrix factorisation and sparse dictionary learning The PCA decomposition is by construction optimal solution to argmin A R n q,h R q p X AH 2 2 under constraint :
Detecting Sparse Structures in Data in Sub-Linear Time: A group testing approach
Detecting Sparse Structures in Data in Sub-Linear Time: A group testing approach Boaz Nadler The Weizmann Institute of Science Israel Joint works with Inbal Horev, Ronen Basri, Meirav Galun and Ery Arias-Castro
Dimension Reduction Methods
Dimension Reduction Methods And Bayesian Machine Learning Marek Petrik 2/28 Previously in Machine Learning How to choose the right features if we have (too) many options Methods: 1. Subset selection 2.
Multipath Matching Pursuit
Multipath Matching Pursuit Submitted to IEEE trans. on Information theory Authors: S. Kwon, J. Wang, and B. Shim Presenter: Hwanchol Jang Multipath is investigated rather than a single path for a greedy
Regularization and Variable Selection via the Elastic Net
p. 1/1 Regularization and Variable Selection via the Elastic Net Hui Zou and Trevor Hastie Journal of Royal Statistical Society, B, 2005 Presenter: Minhua Chen, Nov. 07, 2008 p. 2/1 Agenda Introduction
Symmetric Two Dimensional Linear Discriminant Analysis (2DLDA)
Symmetric Two Dimensional inear Discriminant Analysis (2DDA) Dijun uo, Chris Ding, Heng Huang University of Texas at Arlington 701 S. Nedderman Drive Arlington, TX 76013 dijun.luo@gmail.com, {chqding,
of Orthogonal Matching Pursuit
A Sharp Restricted Isometry Constant Bound of Orthogonal Matching Pursuit Qun Mo arxiv:50.0708v [cs.it] 8 Jan 205 Abstract We shall show that if the restricted isometry constant (RIC) δ s+ (A) of the measurement
Hierarchical Penalization
Hierarchical Penalization Marie Szafransi 1, Yves Grandvalet 1, 2 and Pierre Morizet-Mahoudeaux 1 Heudiasyc 1, UMR CNRS 6599 Université de Technologie de Compiègne BP 20529, 60205 Compiègne Cedex, France
Permutation-invariant regularization of large covariance matrices. Liza Levina
Liza Levina Permutation-invariant covariance regularization 1/42 Permutation-invariant regularization of large covariance matrices Liza Levina Department of Statistics University of Michigan Joint work
A Continuation Approach to Estimate a Solution Path of Mixed L2-L0 Minimization Problems
A Continuation Approach to Estimate a Solution Path of Mixed L2-L Minimization Problems Junbo Duan, Charles Soussen, David Brie, Jérôme Idier Centre de Recherche en Automatique de Nancy Nancy-University,
GROUP-SPARSE SUBSPACE CLUSTERING WITH MISSING DATA
GROUP-SPARSE SUBSPACE CLUSTERING WITH MISSING DATA D Pimentel-Alarcón 1, L Balzano 2, R Marcia 3, R Nowak 1, R Willett 1 1 University of Wisconsin - Madison, 2 University of Michigan - Ann Arbor, 3 University
Block-Sparse Recovery via Convex Optimization
1 Block-Sparse Recovery via Convex Optimization Ehsan Elhamifar, Student Member, IEEE, and René Vidal, Senior Member, IEEE arxiv:11040654v3 [mathoc 13 Apr 2012 Abstract Given a dictionary that consists
Recent Advances in Structured Sparse Models
Recent Advances in Structured Sparse Models Julien Mairal Willow group - INRIA - ENS - Paris 21 September 2010 LEAR seminar At Grenoble, September 21 st, 2010 Julien Mairal Recent Advances in Structured
COMPARATIVE ANALYSIS OF ORTHOGONAL MATCHING PURSUIT AND LEAST ANGLE REGRESSION
COMPARATIVE ANALYSIS OF ORTHOGONAL MATCHING PURSUIT AND LEAST ANGLE REGRESSION By Mazin Abdulrasool Hameed A THESIS Submitted to Michigan State University in partial fulfillment of the requirements for
On the Projection Matrices Influence in the Classification of Compressed Sensed ECG Signals
On the Projection Matrices Influence in the Classification of Compressed Sensed ECG Signals Monica Fira, Liviu Goras Institute of Computer Science Romanian Academy Iasi, Romania Liviu Goras, Nicolae Cleju,
SPARSE signal representations have gained popularity in recent
6958 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 57, NO. 10, OCTOBER 2011 Blind Compressed Sensing Sivan Gleichman and Yonina C. Eldar, Senior Member, IEEE Abstract The fundamental principle underlying
Automatic Subspace Learning via Principal Coefficients Embedding
IEEE TRANSACTIONS ON CYBERNETICS 1 Automatic Subspace Learning via Principal Coefficients Embedding Xi Peng, Jiwen Lu, Senior Member, IEEE, Zhang Yi, Fellow, IEEE and Rui Yan, Member, IEEE, arxiv:1411.4419v5
A new method on deterministic construction of the measurement matrix in compressed sensing
A new method on deterministic construction of the measurement matrix in compressed sensing Qun Mo 1 arxiv:1503.01250v1 [cs.it] 4 Mar 2015 Abstract Construction on the measurement matrix A is a central
Generalized Power Method for Sparse Principal Component Analysis
Generalized Power Method for Sparse Principal Component Analysis Peter Richtárik CORE/INMA Catholic University of Louvain Belgium VOCAL 2008, Veszprém, Hungary CORE Discussion Paper #2008/70 joint work
The lasso an l 1 constraint in variable selection
The lasso algorithm is due to Tibshirani who realised the possibility for variable selection of imposing an l 1 norm bound constraint on the variables in least squares models and then tuning the model
Pre-weighted Matching Pursuit Algorithms for Sparse Recovery
Journal of Information & Computational Science 11:9 (214) 2933 2939 June 1, 214 Available at http://www.joics.com Pre-weighted Matching Pursuit Algorithms for Sparse Recovery Jingfei He, Guiling Sun, Jie
Data Mining und Maschinelles Lernen
Data Mining und Maschinelles Lernen Ensemble Methods Bias-Variance Trade-off Basic Idea of Ensembles Bagging Basic Algorithm Bagging with Costs Randomization Random Forests Boosting Stacking Error-Correcting
Sparse Approximation of Signals with Highly Coherent Dictionaries
Sparse Approximation of Signals with Highly Coherent Dictionaries Bishnu P. Lamichhane and Laura Rebollo-Neira b.p.lamichhane@aston.ac.uk, rebollol@aston.ac.uk Support from EPSRC (EP/D062632/1) is acknowledged
GREEDY SIGNAL RECOVERY REVIEW
GREEDY SIGNAL RECOVERY REVIEW DEANNA NEEDELL, JOEL A. TROPP, ROMAN VERSHYNIN Abstract. The two major approaches to sparse recovery are L 1-minimization and greedy methods. Recently, Needell and Vershynin
Compressed Sensing Meets Machine Learning - Classification of Mixture Subspace Models via Sparse Representation
Compressed Sensing Meets Machine Learning - Classification of Mixture Subspace Models via Sparse Representation Allen Y Yang Feb 25, 2008 UC Berkeley What is Sparsity Sparsity A
EXAM IN STATISTICAL MACHINE LEARNING STATISTISK MASKININLÄRNING
EXAM IN STATISTICAL MACHINE LEARNING STATISTISK MASKININLÄRNING DATE AND TIME: August 30, 2018, 14.00 19.00 RESPONSIBLE TEACHER: Niklas Wahlström NUMBER OF PROBLEMS: 5 AIDING MATERIAL: Calculator, mathematical
Riemannian Metric Learning for Symmetric Positive Definite Matrices
CMSC 88J: Linear Subspaces and Manifolds for Computer Vision and Machine Learning Riemannian Metric Learning for Symmetric Positive Definite Matrices Raviteja Vemulapalli Guide: Professor David W. Jacobs
Strengthened Sobolev inequalities for a random subspace of functions
Strengthened Sobolev inequalities for a random subspace of functions Rachel Ward University of Texas at Austin April 2013 2 Discrete Sobolev inequalities Proposition (Sobolev inequality for discrete images)
Generalized Orthogonal Matching Pursuit- A Review and Some
Generalized Orthogonal Matching Pursuit- A Review and Some New Results Department of Electronics and Electrical Communication Engineering Indian Institute of Technology, Kharagpur, INDIA Table of Contents
Structured matrix factorizations. Example: Eigenfaces
Structured matrix factorizations Example: Eigenfaces An extremely large variety of interesting and important problems in machine learning can be formulated as: Given a matrix, find a matrix and a matrix
Applied Machine Learning for Biomedical Engineering. Enrico Grisan
Applied Machine Learning for Biomedical Engineering Enrico Grisan enrico.grisan@dei.unipd.it Data representation To find a representation that approximates elements of a signal class with a linear combination
Analysis of Robust PCA via Local Incoherence
Analysis of Robust PCA via Local Incoherence Huishuai Zhang Department of EECS Syracuse University Syracuse, NY 3244 hzhan23@syr.edu Yi Zhou Department of EECS Syracuse University Syracuse, NY 3244 yzhou35@syr.edu
SEQUENTIAL SUBSPACE FINDING: A NEW ALGORITHM FOR LEARNING LOW-DIMENSIONAL LINEAR SUBSPACES.
SEQUENTIAL SUBSPACE FINDING: A NEW ALGORITHM FOR LEARNING LOW-DIMENSIONAL LINEAR SUBSPACES Mostafa Sadeghi a, Mohsen Joneidi a, Massoud Babaie-Zadeh a, and Christian Jutten b a Electrical Engineering Department,
arxiv: v1 [math.oc] 30 Apr 2016
![arxiv: v1 [math.oc] 30 Apr 2016](/thumbs/72/67218075.jpg "arxiv: v1 [math.oc] 30 Apr 2016")
The Homotopy Method Revisited: Computing Solution Paths of l -Regularized Problems Björn Bringmann, Daniel Cremers, Felix Krahmer, Michael Möller arxiv:605.0007v [math.oc] 30 Apr 206 May 3, 206 Abstract
Recap from previous lecture
Recap from previous lecture Learning is using past experience to improve future performance. Different types of learning: supervised unsupervised reinforcement active online... For a machine, experience
Discussion of Least Angle Regression
Discussion of Least Angle Regression David Madigan Rutgers University & Avaya Labs Research Piscataway, NJ 08855 madigan@stat.rutgers.edu Greg Ridgeway RAND Statistics Group Santa Monica, CA 90407-2138
Sparse analysis Lecture III: Dictionary geometry and greedy algorithms
Sparse analysis Lecture III: Dictionary geometry and greedy algorithms Anna C. Gilbert Department of Mathematics University of Michigan Intuition from ONB Key step in algorithm: r, ϕ j = x c i ϕ i, ϕ j
Effective Linear Discriminant Analysis for High Dimensional, Low Sample Size Data
Effective Linear Discriant Analysis for High Dimensional, Low Sample Size Data Zhihua Qiao, Lan Zhou and Jianhua Z. Huang Abstract In the so-called high dimensional, low sample size (HDLSS) settings, LDA
A New Perspective on Boosting in Linear Regression via Subgradient Optimization and Relatives
A New Perspective on Boosting in Linear Regression via Subgradient Optimization and Relatives Paul Grigas May 25, 2016 1 Boosting Algorithms in Linear Regression Boosting [6, 9, 12, 15, 16] is an extremely
Deep Learning: Approximation of Functions by Composition
Deep Learning: Approximation of Functions by Composition Zuowei Shen Department of Mathematics National University of Singapore Outline 1 A brief introduction of approximation theory 2 Deep learning: approximation
Simultaneous Sparsity
Simultaneous Sparsity Joel A. Tropp Anna C. Gilbert Martin J. Strauss {jtropp annacg martinjs}@umich.edu Department of Mathematics The University of Michigan 1 Simple Sparse Approximation Work in the d-dimensional,