Fast Nonnegative Matrix Factorization with Rank-one ADMM
|
|
- Oswald Perry
- 6 years ago
- Views:
Transcription
1 Fast Nonnegative Matrix Factorization with Rank-one Dongjin Song, David A. Meyer, Martin Renqiang Min, Department of ECE, UCSD, La Jolla, CA, Department of Mathematics, UCSD, La Jolla, CA, NEC Labs America, Princeton, NJ Abstract Nonnegative matrix factorization NMF, which aims to approximate a data matrix with two nonnegative low rank matrix factors, is a popular dimensionality reduction and clustering technique. Due to the non-convex formulation and the nonnegativity constraints over the two low rank matrix factors with rank r > 0, it is often difficult to solve NMF efficiently and accurately. Recently, the alternating direction method of multiplier was shown to be more accurate and efficient than classical approaches such as the multiplicative update rule or alternating least square. Nevertheless, the computation of is proportional to the cube of the rank r because of the underlying matrix inverse problem and thus may be inefficient when r is relatively large. In this paper, we propose a rank-one to address this problem. In each step, we search for a rank-one solution of NMF based upon and utilize greedy search to obtain the low rank matrix factors. In this way, rank-one avoids the matrix inverse problem and the computation is only linearly proportional to r. Thorough empirical studies demonstrate that rank-one is more efficient and accurate than baseline approaches. Introduction In the past decade, nonnegative matrix factorization NMF [5][6] and its extensions have been widely applied for various applications, e.g., face recognition [][3], scene classification [0], social network analysis [][], bioinformatics [9], etc. Essentially, NMF aims to find two nonnegative low rank matrix factors U R n r and V R r d so as to reconstruct a data matrix X R n d of n samples and d features, i.e., min U,V X UV F s.t. U 0, V 0 where r denotes rank, F is the Frobenius norm, and the inequalities are element-wise. Since NMF is non-convex and U and V are constrained to be nonnegative, it is difficult to solve NMF accurately and efficiently. To address this issue, Lee and Seung [6] developed multiplicative update rule which is guaranteed to obtain a locally optimal solution. Since converges very slowly, alternating least square [8] and projected gradient method [7] were subsequently developed. Recently, alternating direction method of multiplier [] has shown its superiority over and with respect to both reconstruction accuracy and efficiency [5][4][3]. Nevertheless, since the computation of involves a matrix inverse problem, the complexity of is proportional to the cube of the rank r of two low rank matrix factors and may be inefficient when r is relatively large. To resolve this issue, we propose rank-one in this paper. The main idea
2 is to find a rank-one solution of NMF based upon and utilize greedy search to obtain the low rank matrix factors. In this way, rank-one avoids the matrix inverse problem and the computation is only linearly proportional to r. Related Work In this section, we briefly introduce three representative approaches for optimizating NMF, i.e., multiplicative update rule [6], alternating least square [8], and alternating direction method of multiplier []. Notations: Given a matrix X R n d of n rows and d columns, we use x j R n to denote its j-th column, use x i R d to denote its i-th row, and use X ij to denote the entry in i-th row and j-th column of X.. Multiplicative Update Rule proposed by Lee and Seung [6] is the most popular method for NMF and can be written as: XV T ik U ik U ik UV V T ik and U T X kj V kj V kj U T. UV kj 3 By using with nonnegative initializations of U and V, both U and V will remain nonnegative throughout iterations. Since converges slowly in many real world applications, [8] can be utilized as the substitution.. Alternating Least Square developed by Paatero and Taper [8] minimizes the least square cost function in Eq. with respect to U and V, iteratively. The procedure is given as and U = P + XV T V V T 4 V = P + U T U U T X, 5 where denotes pseudo-inverse and P + projects the matrix onto a nonnegative set, i.e., P + U = maxu, 0. Although has shown its efficiency for NMF, recent research indicates that alternating direction method of multiplier [] is more effective and efficient than [5][4]..3 Alternating Direction Method of Multipliers The alternating direction method of multipliers [] aims to solve convex optimization problems by splitting them into smaller pieces such that each part of them is easier to handle. It could also be used for non-convex problems such as NMF [5][4][3]. Specifically, introduces two auxiliary variables S and T and considers the following equivalent formulation: min U,V,S,T X UV F 6 s.t. U S = 0, V T = 0, S 0, T 0. The augmented Lagrangian function of Eq. 6 is given as: LU, V, S, T, Λ, Π = X UV F + Λ, U S + Π, V T + ρ U S F + ρ V T F 7 where Λ R n r and Π R r d are Lagrange multipliers, is the matrix inner product, and ρ > 0 is the penalty parameter for the constraints. By minimizing L with respect to U, V, S, T, Λ, and Π, we can obtain the closed form solution for each step as shown in appendix Algorithm. Note that the main computations in classical are the inversion of an r r matrix and XV T or U T X. When r is relatively large e.g., 500 or 000 in some real world applications, the matrix inverse problem will consume substantial time at each step and thus will converge slowly. To resolve this issue, rank-one is developed.
3 Algorithm : Rank-one for NMF Input: X R n d, u i+ R n, v i+ R d, s i+ R n, t i+ R d, p R n, q R d, ρ Output: U and V : Set X = X; : Calculate u 0, v 0 based upon 3: For i = 0 to r, 4: Set k = 0 index of iteration 5: X = X uiv i 6: Repeat: [ T ] [ T ] 7: u i+k + = X v k i+ + ρsi+k pk / v i+ k v i+ k + ρ [ T ] [ T ] 8: v i+ k + = u i+k + X + ρt i+ k qk / u i+k + ui+k + + ρ 9: s i+k + = P + u i+k + + pk/ρ 0: t i+ k + = P + v i+ k + + qk/ρ : pk + = pk + ρ u i+k + s i+k + : qk + = qk + ρ v i+ k + t i+ k + 3: k = k + 4: Until convergence. 5: U = [U, u i+k + ], V = [V ; v i+ k + ], S = [S, s i+k + ], T = [T ; t i+ k + ]. 6: End for. 3 Rank-one for NMF In rank-one, we successively search for rank-one solution to NMF based upon and utilize greedy search to obtain the low rank matrix factors U R n r and V R r d. For initialization i = 0, we can adopt classical to obtain a rank-one solution of NMF. For the i + step, since U R n i and V R i d are already known and can be written as concatenations of column vectors U = [u,..., u i ] and row vectors V = [v ;...; v i ] respectively, rank-one can be formulated as: min X [u,..., u i, u i+ ][v ;...; v r ; v i+ ] u i+,v i+,s i+,t i+ F 8 s.t. u i+ s i+ = 0, v i+ t i+ = 0, s i+ 0, t i+ 0. Since we have known U = [u,..., u i ] and V = [v ;...; v i ], Eq. 8 can be simplified as: min u i+,v i+,s i+,t i+ X u i+ v i+ F s.t. u i+ s i+ = 0, v i+ t i+ = 0, s i+ 0, t i+ 0, where X = X UV. The augmented Lagrangian of Eq. 0 can be written as: Lu i+, v i+, s i+, t i+, p, q = X u i+ v i+ + p, u i+ s i+ + q, v i+ t i+ + F ρ u i+ s i+ + ρ vi+ t i+. 0 where p R n and q R d are Lagrangian multipliers, and ρ > 0 is the penalty parameter for the constraints. By minimizing L with respect to u i+, v i+, s i+, t i+, p, and q, we can obtain the closed form solution of each step as shown in Algorithm. In Algorithm, u i+ and v i+ are randomly initialized. s i+, t i+, p, and q are set to zero vectors of appropriate lengths. The stopping criterion for the inner loop of rank-one is met if the objective in Eq. 8 does not improve relative to a tolerance value. In the inner loop, the main computation are the u i+ k + updating and v i+ k + updating with computational complexity of Ond + and On + d, respectively. Assuming that on 9 3
4 Time seconds Rank r a Runing time seconds vs. rank Accuracy Rank r b Accuracy vs. rank NMI Rank r c NMI vs. rank Figure : Experiment results on UMIST dataset. Time seconds Accuracy NMI Rank r a Runing time seconds vs. rank 0.45 Rank r b Accuracy vs. rank Rank r c NMI vs. rank Figure : Experiment results on Coil0 dataset. average the inner loop takes k iterations to achieve convergence, the total computational complexity of rank-one will be Ondrk + nrk + drk which eliminates the matrix inverse problem in classical as shown in Algorithm of Appendix. 4 Experiments In this section, we evaluate the effectiveness and efficiency of our proposed approach by comparing with three baseline approaches, i.e.,,, and. In particular, we implement all four approaches over two publicly available datasets, i.e., UMIST and Coil0. UMIST dataset contains 575 size of face images of 0 different persons. Coil0 dataset includes 440 size of 3 3 object images of 0 different classes. The experiment results over UMIST and Coil0 are shown in Figure and Figure, respectively. Figure a and a show the running time of each approach for convergence versus the rank r. we observe that when the rank r varies from 00 to 000, rank-one generally converges faster than other approaches, especially when r is large. This is because the computation of and is proportional to the cube of the rank due to the underlying matrix inverse problem; while the computation of rank-one is only linearly proportional to the cube of the rank. Moreover, we evaluate effectiveness of the low rank representations of four approaches based upon clustering accuracy and normalized mutual information NMI in Figure b, Figure c, Figure b, and Figure c. We observe that rank-one generally achieves better accuracy and NMI than the other approaches. This may because rank-one tends to yield more sparse low rank representations than other approaches. 5 Conclusion In this paper, we proposed a rank-one alternating direction method of multiplier for nonnegative matrix factorization NMF. In each step, rank-one seeks for a rank-one solution of NMF based upon and utilizes greedy search to obtain low rank matrix factors. In this way, rank-one avoids the underlying matrix inverse problem and the computation is only linearly proportional to the rank r. We conducted empirical studies based upon two publicly available datasets, i.e., UMIST and Coil0. Our results demonstrated that rank-one is more efficient and effective than,, and traditional. 4
5 References [] Boyd, S., Parikh, S., Chu, E., Peleato, B., and Eckstein, J. 0 Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends Rin Machine Learning, 3: -. [] Guan, N., Tao, D., Luo, Z., Yuan, B. 0 Manifold regularized discriminative nonnegative matrix factorization with fast gradient descent. IEEE Transactions on Image Processing, vol. 0, no. 7, pp [3] Guan, N., Tao, D., Luo, and Shawe-Taylor, J. 0 MahNMF: Manhattan non-negative matrix factorization. arxiv preprint arxiv: [4] Kim, J. and Park, H. 0 Fast nonnegative matrix factorization: An active-set-like method and comparisons. SIAM Journal on Scientific Computing 336: 36-38, 0. [5] Lee, D. and Seung, H. 999 Learning the parts of objects by non-negative matrix factorization. Nature 40: [6] Lee, D. and Seung, H. 00 Algorithms for non-negative matrix factorization. Advances in Neural Information Processing Systems 3, pp Cambridge, MA: MIT Press. [7] Lin, C.-J. 007 Projected gradient methods for nonnegative matrix factorization. Neural Computation 90: [8] Paatero, P. and Tapper, U. 994 Positive matrix factorization: a non-negative factor model with optimal utilization of error estimates of data values. Environmetrics 5-6. [9] Min, M. R., Ning, X., Cheng, C., and Gerstein, M. 04 Interpretable sparse high-order Boltzmann machines. The 7th International Conference on Artificial Intelligence and Statistics. [0] Song, D. and Tao, D. 00 Biologically inspired feature manifold for scene classification. IEEE Transactions on Image Processing, vol. 9, no., pp [] Song, D. and David, A. M. 04 A model of consistent node types in signed directed social networks. IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining ASONAM, pp [] Song, D. and David, A. M. 04 Recommending positive links in signed social networks by optimizing a generalized AUC. The Twenty-Ninth AAAI Conference on Artificial Intelligence AAAI [3] Sun, L. and Févotte, C., 04 Alternating direction method of multipliers for non-negative matrix factorization with the beta-divergence. IEEE International Conference on Acoustics, Speech, and Signal Processing. [4] Xu, Y., Yin, W., Wen, Z., and Zhang, Y. 0 An alternating direction algorithm for matrix completion with nonnegative factors. Frontiers of Mathematics in China, 7: [5] Zhang, Y. 00. An alternating direction algorithm for nonnegative matrix factorization. Technical Report, Rice University. 5
6 Appendix Algorithm : for NMF : Input: X R n d, U R n r, V R r d, S R n r, T R r d, ρ : Output: U and V 3: Set k = 0 index of iteration 4: Repeat: 5: Uk + = XV k T + ρsk Λk V kv k T + ρi 6: V k + = Uk + T Uk + + ρi 7: Sk + = P + Uk + + Λk/ρ 8: T k + = P + V k + + Πk/ρ 9: Λk + = Λk + ρ Uk + Sk + 0: Πk + = Πk + ρ V k + T k + : k = k +. : Until convergence. Uk + T X + ρt k Πk In Algorithm, U and V are randomly initialized. S, T, Λ, and Π are set to zero matrices of appropriate sizes. The stopping criterion for is met if the objective in Eq. 6 does not improve relative to a tolerance value. In each step, the main computation are Uk + updating and V k + updating, which are Ondr +dr +r 3 and Ondr + nr + r 3, respectively. Assuming takes k iteration to achieve convergence, the total computational complexity for will be Ondrk + nrk + drk + kr 3. 6
Factor Analysis (FA) Non-negative Matrix Factorization (NMF) CSE Artificial Intelligence Grad Project Dr. Debasis Mitra
Factor Analysis (FA) Non-negative Matrix Factorization (NMF) CSE 5290 - Artificial Intelligence Grad Project Dr. Debasis Mitra Group 6 Taher Patanwala Zubin Kadva Factor Analysis (FA) 1. Introduction Factor
More informationONP-MF: An Orthogonal Nonnegative Matrix Factorization Algorithm with Application to Clustering
ONP-MF: An Orthogonal Nonnegative Matrix Factorization Algorithm with Application to Clustering Filippo Pompili 1, Nicolas Gillis 2, P.-A. Absil 2,andFrançois Glineur 2,3 1- University of Perugia, Department
More informationA Bregman alternating direction method of multipliers for sparse probabilistic Boolean network problem
A Bregman alternating direction method of multipliers for sparse probabilistic Boolean network problem Kangkang Deng, Zheng Peng Abstract: The main task of genetic regulatory networks is to construct a
More informationOrthogonal Nonnegative Matrix Factorization: Multiplicative Updates on Stiefel Manifolds
Orthogonal Nonnegative Matrix Factorization: Multiplicative Updates on Stiefel Manifolds Jiho Yoo and Seungjin Choi Department of Computer Science Pohang University of Science and Technology San 31 Hyoja-dong,
More informationPreconditioning via Diagonal Scaling
Preconditioning via Diagonal Scaling Reza Takapoui Hamid Javadi June 4, 2014 1 Introduction Interior point methods solve small to medium sized problems to high accuracy in a reasonable amount of time.
More informationFantope Regularization in Metric Learning
Fantope Regularization in Metric Learning CVPR 2014 Marc T. Law (LIP6, UPMC), Nicolas Thome (LIP6 - UPMC Sorbonne Universités), Matthieu Cord (LIP6 - UPMC Sorbonne Universités), Paris, France Introduction
More informationNonnegative Matrix Factorization Clustering on Multiple Manifolds
Proceedings of the Twenty-Fourth AAAI Conference on Artificial Intelligence (AAAI-10) Nonnegative Matrix Factorization Clustering on Multiple Manifolds Bin Shen, Luo Si Department of Computer Science,
More informationNote on Algorithm Differences Between Nonnegative Matrix Factorization And Probabilistic Latent Semantic Indexing
Note on Algorithm Differences Between Nonnegative Matrix Factorization And Probabilistic Latent Semantic Indexing 1 Zhong-Yuan Zhang, 2 Chris Ding, 3 Jie Tang *1, Corresponding Author School of Statistics,
More informationarxiv: v3 [cs.lg] 18 Mar 2013
Hierarchical Data Representation Model - Multi-layer NMF arxiv:1301.6316v3 [cs.lg] 18 Mar 2013 Hyun Ah Song Department of Electrical Engineering KAIST Daejeon, 305-701 hyunahsong@kaist.ac.kr Abstract Soo-Young
More informationarxiv: v1 [math.oc] 23 May 2017
A DERANDOMIZED ALGORITHM FOR RP-ADMM WITH SYMMETRIC GAUSS-SEIDEL METHOD JINCHAO XU, KAILAI XU, AND YINYU YE arxiv:1705.08389v1 [math.oc] 23 May 2017 Abstract. For multi-block alternating direction method
More information2 Regularized Image Reconstruction for Compressive Imaging and Beyond
EE 367 / CS 448I Computational Imaging and Display Notes: Compressive Imaging and Regularized Image Reconstruction (lecture ) Gordon Wetzstein gordon.wetzstein@stanford.edu This document serves as a supplement
More informationAN ENHANCED INITIALIZATION METHOD FOR NON-NEGATIVE MATRIX FACTORIZATION. Liyun Gong 1, Asoke K. Nandi 2,3 L69 3BX, UK; 3PH, UK;
213 IEEE INERNAIONAL WORKSHOP ON MACHINE LEARNING FOR SIGNAL PROCESSING, SEP. 22 25, 213, SOUHAMPON, UK AN ENHANCED INIIALIZAION MEHOD FOR NON-NEGAIVE MARIX FACORIZAION Liyun Gong 1, Asoke K. Nandi 2,3
More informationAn efficient ADMM algorithm for high dimensional precision matrix estimation via penalized quadratic loss
An efficient ADMM algorithm for high dimensional precision matrix estimation via penalized quadratic loss arxiv:1811.04545v1 [stat.co] 12 Nov 2018 Cheng Wang School of Mathematical Sciences, Shanghai Jiao
More informationPreserving Privacy in Data Mining using Data Distortion Approach
Preserving Privacy in Data Mining using Data Distortion Approach Mrs. Prachi Karandikar #, Prof. Sachin Deshpande * # M.E. Comp,VIT, Wadala, University of Mumbai * VIT Wadala,University of Mumbai 1. prachiv21@yahoo.co.in
More informationNonnegative Matrix Factorization
Nonnegative Matrix Factorization Seungjin Choi Department of Computer Science and Engineering Pohang University of Science and Technology 77 Cheongam-ro, Nam-gu, Pohang 37673, Korea seungjin@postech.ac.kr
More informationSimplicial Nonnegative Matrix Tri-Factorization: Fast Guaranteed Parallel Algorithm
Simplicial Nonnegative Matrix Tri-Factorization: Fast Guaranteed Parallel Algorithm Duy-Khuong Nguyen 13, Quoc Tran-Dinh 2, and Tu-Bao Ho 14 1 Japan Advanced Institute of Science and Technology, Japan
More informationNon-Negative Matrix Factorization with Quasi-Newton Optimization
Non-Negative Matrix Factorization with Quasi-Newton Optimization Rafal ZDUNEK, Andrzej CICHOCKI Laboratory for Advanced Brain Signal Processing BSI, RIKEN, Wako-shi, JAPAN Abstract. Non-negative matrix
More informationNON-NEGATIVE SPARSE CODING
NON-NEGATIVE SPARSE CODING Patrik O. Hoyer Neural Networks Research Centre Helsinki University of Technology P.O. Box 9800, FIN-02015 HUT, Finland patrik.hoyer@hut.fi To appear in: Neural Networks for
More informationNonlinear Support Vector Machines through Iterative Majorization and I-Splines
Nonlinear Support Vector Machines through Iterative Majorization and I-Splines P.J.F. Groenen G. Nalbantov J.C. Bioch July 9, 26 Econometric Institute Report EI 26-25 Abstract To minimize the primal support
More informationIterative Laplacian Score for Feature Selection
Iterative Laplacian Score for Feature Selection Linling Zhu, Linsong Miao, and Daoqiang Zhang College of Computer Science and echnology, Nanjing University of Aeronautics and Astronautics, Nanjing 2006,
More informationA Customized ADMM for Rank-Constrained Optimization Problems with Approximate Formulations
A Customized ADMM for Rank-Constrained Optimization Problems with Approximate Formulations Chuangchuang Sun and Ran Dai Abstract This paper proposes a customized Alternating Direction Method of Multipliers
More informationc Springer, Reprinted with permission.
Zhijian Yuan and Erkki Oja. A FastICA Algorithm for Non-negative Independent Component Analysis. In Puntonet, Carlos G.; Prieto, Alberto (Eds.), Proceedings of the Fifth International Symposium on Independent
More informationA Brief Overview of Practical Optimization Algorithms in the Context of Relaxation
A Brief Overview of Practical Optimization Algorithms in the Context of Relaxation Zhouchen Lin Peking University April 22, 2018 Too Many Opt. Problems! Too Many Opt. Algorithms! Zero-th order algorithms:
More informationGroup Sparse Non-negative Matrix Factorization for Multi-Manifold Learning
LIU, LU, GU: GROUP SPARSE NMF FOR MULTI-MANIFOLD LEARNING 1 Group Sparse Non-negative Matrix Factorization for Multi-Manifold Learning Xiangyang Liu 1,2 liuxy@sjtu.edu.cn Hongtao Lu 1 htlu@sjtu.edu.cn
More informationMulti-Task Clustering using Constrained Symmetric Non-Negative Matrix Factorization
Multi-Task Clustering using Constrained Symmetric Non-Negative Matrix Factorization Samir Al-Stouhi Chandan K. Reddy Abstract Researchers have attempted to improve the quality of clustering solutions through
More informationCOMS 4721: Machine Learning for Data Science Lecture 18, 4/4/2017
COMS 4721: Machine Learning for Data Science Lecture 18, 4/4/2017 Prof. John Paisley Department of Electrical Engineering & Data Science Institute Columbia University TOPIC MODELING MODELS FOR TEXT DATA
More informationMULTIPLICATIVE ALGORITHM FOR CORRENTROPY-BASED NONNEGATIVE MATRIX FACTORIZATION
MULTIPLICATIVE ALGORITHM FOR CORRENTROPY-BASED NONNEGATIVE MATRIX FACTORIZATION Ehsan Hosseini Asl 1, Jacek M. Zurada 1,2 1 Department of Electrical and Computer Engineering University of Louisville, Louisville,
More informationSparseness Constraints on Nonnegative Tensor Decomposition
Sparseness Constraints on Nonnegative Tensor Decomposition Na Li nali@clarksonedu Carmeliza Navasca cnavasca@clarksonedu Department of Mathematics Clarkson University Potsdam, New York 3699, USA Department
More informationCONVOLUTIVE NON-NEGATIVE MATRIX FACTORISATION WITH SPARSENESS CONSTRAINT
CONOLUTIE NON-NEGATIE MATRIX FACTORISATION WITH SPARSENESS CONSTRAINT Paul D. O Grady Barak A. Pearlmutter Hamilton Institute National University of Ireland, Maynooth Co. Kildare, Ireland. ABSTRACT Discovering
More informationNonnegative Matrix Factorization with Toeplitz Penalty
Journal of Informatics and Mathematical Sciences Vol. 10, Nos. 1 & 2, pp. 201 215, 2018 ISSN 0975-5748 (online); 0974-875X (print) Published by RGN Publications http://dx.doi.org/10.26713/jims.v10i1-2.851
More informationApproximating the Partition Function by Deleting and then Correcting for Model Edges (Extended Abstract)
Approximating the Partition Function by Deleting and then Correcting for Model Edges (Extended Abstract) Arthur Choi and Adnan Darwiche Computer Science Department University of California, Los Angeles
More informationContraction Methods for Convex Optimization and Monotone Variational Inequalities No.16
XVI - 1 Contraction Methods for Convex Optimization and Monotone Variational Inequalities No.16 A slightly changed ADMM for convex optimization with three separable operators Bingsheng He Department of
More informationGraph-Laplacian PCA: Closed-form Solution and Robustness
2013 IEEE Conference on Computer Vision and Pattern Recognition Graph-Laplacian PCA: Closed-form Solution and Robustness Bo Jiang a, Chris Ding b,a, Bin Luo a, Jin Tang a a School of Computer Science and
More informationBlock Coordinate Descent for Regularized Multi-convex Optimization
Block Coordinate Descent for Regularized Multi-convex Optimization Yangyang Xu and Wotao Yin CAAM Department, Rice University February 15, 2013 Multi-convex optimization Model definition Applications Outline
More informationEUSIPCO
EUSIPCO 2013 1569741067 CLUSERING BY NON-NEGAIVE MARIX FACORIZAION WIH INDEPENDEN PRINCIPAL COMPONEN INIIALIZAION Liyun Gong 1, Asoke K. Nandi 2,3 1 Department of Electrical Engineering and Electronics,
More informationSINGLE CHANNEL SPEECH MUSIC SEPARATION USING NONNEGATIVE MATRIX FACTORIZATION AND SPECTRAL MASKS. Emad M. Grais and Hakan Erdogan
SINGLE CHANNEL SPEECH MUSIC SEPARATION USING NONNEGATIVE MATRIX FACTORIZATION AND SPECTRAL MASKS Emad M. Grais and Hakan Erdogan Faculty of Engineering and Natural Sciences, Sabanci University, Orhanli
More informationSparse Gaussian conditional random fields
Sparse Gaussian conditional random fields Matt Wytock, J. ico Kolter School of Computer Science Carnegie Mellon University Pittsburgh, PA 53 {mwytock, zkolter}@cs.cmu.edu Abstract We propose sparse Gaussian
More informationNMF WITH SPECTRAL AND TEMPORAL CONTINUITY CRITERIA FOR MONAURAL SOUND SOURCE SEPARATION. Julian M. Becker, Christian Sohn and Christian Rohlfing
NMF WITH SPECTRAL AND TEMPORAL CONTINUITY CRITERIA FOR MONAURAL SOUND SOURCE SEPARATION Julian M. ecker, Christian Sohn Christian Rohlfing Institut für Nachrichtentechnik RWTH Aachen University D-52056
More informationRank Selection in Low-rank Matrix Approximations: A Study of Cross-Validation for NMFs
Rank Selection in Low-rank Matrix Approximations: A Study of Cross-Validation for NMFs Bhargav Kanagal Department of Computer Science University of Maryland College Park, MD 277 bhargav@cs.umd.edu Vikas
More informationNon-Negative Matrix Factorization
Chapter 3 Non-Negative Matrix Factorization Part : Introduction & computation Motivating NMF Skillicorn chapter 8; Berry et al. (27) DMM, summer 27 2 Reminder A T U Σ V T T Σ, V U 2 Σ 2,2 V 2.8.6.6.3.6.5.3.6.3.6.4.3.6.4.3.3.4.5.3.5.8.3.8.3.3.5
More informationQuick Introduction to Nonnegative Matrix Factorization
Quick Introduction to Nonnegative Matrix Factorization Norm Matloff University of California at Davis 1 The Goal Given an u v matrix A with nonnegative elements, we wish to find nonnegative, rank-k matrices
More informationLinearized Alternating Direction Method with Adaptive Penalty for Low-Rank Representation
Linearized Alternating Direction Method with Adaptive Penalty for Low-Rank Representation Zhouchen Lin Visual Computing Group Microsoft Research Asia Risheng Liu Zhixun Su School of Mathematical Sciences
More informationThe Alternating Direction Method of Multipliers
The Alternating Direction Method of Multipliers With Adaptive Step Size Selection Peter Sutor, Jr. Project Advisor: Professor Tom Goldstein December 2, 2015 1 / 25 Background The Dual Problem Consider
More informationGeneralized Nonnegative Matrix Approximations with Bregman Divergences
Generalized Nonnegative Matrix Approximations with Bregman Divergences Inderjit S. Dhillon Suvrit Sra Dept. of Computer Sciences The Univ. of Texas at Austin Austin, TX 78712. {inderjit,suvrit}@cs.utexas.edu
More informationDecentralized Quadratically Approximated Alternating Direction Method of Multipliers
Decentralized Quadratically Approimated Alternating Direction Method of Multipliers Aryan Mokhtari Wei Shi Qing Ling Alejandro Ribeiro Department of Electrical and Systems Engineering, University of Pennsylvania
More informationNonnegative Tensor Factorization using a proximal algorithm: application to 3D fluorescence spectroscopy
Nonnegative Tensor Factorization using a proximal algorithm: application to 3D fluorescence spectroscopy Caroline Chaux Joint work with X. Vu, N. Thirion-Moreau and S. Maire (LSIS, Toulon) Aix-Marseille
More informationAdaptive Kernel Principal Component Analysis With Unsupervised Learning of Kernels
Adaptive Kernel Principal Component Analysis With Unsupervised Learning of Kernels Daoqiang Zhang Zhi-Hua Zhou National Laboratory for Novel Software Technology Nanjing University, Nanjing 2193, China
More informationRecommendation Systems
Recommendation Systems Popularity Recommendation Systems Predicting user responses to options Offering news articles based on users interests Offering suggestions on what the user might like to buy/consume
More informationAN ALTERNATING MINIMIZATION ALGORITHM FOR NON-NEGATIVE MATRIX APPROXIMATION
AN ALTERNATING MINIMIZATION ALGORITHM FOR NON-NEGATIVE MATRIX APPROXIMATION JOEL A. TROPP Abstract. Matrix approximation problems with non-negativity constraints arise during the analysis of high-dimensional
More information2 GU, ZHOU: NEIGHBORHOOD PRESERVING NONNEGATIVE MATRIX FACTORIZATION graph regularized NMF (GNMF), which assumes that the nearby data points are likel
GU, ZHOU: NEIGHBORHOOD PRESERVING NONNEGATIVE MATRIX FACTORIZATION 1 Neighborhood Preserving Nonnegative Matrix Factorization Quanquan Gu gqq03@mails.tsinghua.edu.cn Jie Zhou jzhou@tsinghua.edu.cn State
More informationContraction Methods for Convex Optimization and Monotone Variational Inequalities No.18
XVIII - 1 Contraction Methods for Convex Optimization and Monotone Variational Inequalities No18 Linearized alternating direction method with Gaussian back substitution for separable convex optimization
More informationParallel and Distributed Stochastic Learning -Towards Scalable Learning for Big Data Intelligence
Parallel and Distributed Stochastic Learning -Towards Scalable Learning for Big Data Intelligence oé LAMDA Group H ŒÆOŽÅ Æ EâX ^ #EâI[ : liwujun@nju.edu.cn Dec 10, 2016 Wu-Jun Li (http://cs.nju.edu.cn/lwj)
More informationRecent Developments of Alternating Direction Method of Multipliers with Multi-Block Variables
Recent Developments of Alternating Direction Method of Multipliers with Multi-Block Variables Department of Systems Engineering and Engineering Management The Chinese University of Hong Kong 2014 Workshop
More informationOn Spectral Basis Selection for Single Channel Polyphonic Music Separation
On Spectral Basis Selection for Single Channel Polyphonic Music Separation Minje Kim and Seungjin Choi Department of Computer Science Pohang University of Science and Technology San 31 Hyoja-dong, Nam-gu
More informationNONNEGATIVE matrix factorization (NMF) is a
Algorithms for Orthogonal Nonnegative Matrix Factorization Seungjin Choi Abstract Nonnegative matrix factorization (NMF) is a widely-used method for multivariate analysis of nonnegative data, the goal
More informationA Purely Geometric Approach to Non-Negative Matrix Factorization
A Purely Geometric Approach to Non-Negative Matrix Factorization Christian Bauckhage B-IT, University of Bonn, Bonn, Germany Fraunhofer IAIS, Sankt Augustin, Germany http://mmprec.iais.fraunhofer.de/bauckhage.html
More informationGROUP-SPARSE SUBSPACE CLUSTERING WITH MISSING DATA
GROUP-SPARSE SUBSPACE CLUSTERING WITH MISSING DATA D Pimentel-Alarcón 1, L Balzano 2, R Marcia 3, R Nowak 1, R Willett 1 1 University of Wisconsin - Madison, 2 University of Michigan - Ann Arbor, 3 University
More informationDivide-and-combine Strategies in Statistical Modeling for Massive Data
Divide-and-combine Strategies in Statistical Modeling for Massive Data Liqun Yu Washington University in St. Louis March 30, 2017 Liqun Yu (WUSTL) D&C Statistical Modeling for Massive Data March 30, 2017
More informationIntroduction to Alternating Direction Method of Multipliers
Introduction to Alternating Direction Method of Multipliers Yale Chang Machine Learning Group Meeting September 29, 2016 Yale Chang (Machine Learning Group Meeting) Introduction to Alternating Direction
More informationSupervised Nonnegative Matrix Factorization to Predict ICU Mortality Risk
Supervised Nonnegative Matrix Factorization to Predict ICU Mortality Risk Guoqing Chao guoqingchao10@gmail.com Chengsheng Mao chengsheng.mao@northwestern.edu Fei Wang Weill Cornell Medicine Cornell University
More informationUses of duality. Geoff Gordon & Ryan Tibshirani Optimization /
Uses of duality Geoff Gordon & Ryan Tibshirani Optimization 10-725 / 36-725 1 Remember conjugate functions Given f : R n R, the function is called its conjugate f (y) = max x R n yt x f(x) Conjugates appear
More informationLinearly-solvable Markov decision problems
Advances in Neural Information Processing Systems 2 Linearly-solvable Markov decision problems Emanuel Todorov Department of Cognitive Science University of California San Diego todorov@cogsci.ucsd.edu
More informationDistributed Inexact Newton-type Pursuit for Non-convex Sparse Learning
Distributed Inexact Newton-type Pursuit for Non-convex Sparse Learning Bo Liu Department of Computer Science, Rutgers Univeristy Xiao-Tong Yuan BDAT Lab, Nanjing University of Information Science and Technology
More informationOn Optimal Frame Conditioners
On Optimal Frame Conditioners Chae A. Clark Department of Mathematics University of Maryland, College Park Email: cclark18@math.umd.edu Kasso A. Okoudjou Department of Mathematics University of Maryland,
More informationRelaxed Majorization-Minimization for Non-smooth and Non-convex Optimization
Relaxed Majorization-Minimization for Non-smooth and Non-convex Optimization Chen Xu 1, Zhouchen Lin 1,2,, Zhenyu Zhao 3, Hongbin Zha 1 1 Key Laboratory of Machine Perception (MOE), School of EECS, Peking
More informationWorst-Case Analysis of the Perceptron and Exponentiated Update Algorithms
Worst-Case Analysis of the Perceptron and Exponentiated Update Algorithms Tom Bylander Division of Computer Science The University of Texas at San Antonio San Antonio, Texas 7849 bylander@cs.utsa.edu April
More informationCoupled Dictionary Learning for Unsupervised Feature Selection
Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence (AAAI-16) Coupled Dictionary Learning for Unsupervised Feature Selection Pengfei Zhu 1, Qinghua Hu 1, Changqing Zhang 1, Wangmeng
More informationData Mining and Matrices
Data Mining and Matrices 6 Non-Negative Matrix Factorization Rainer Gemulla, Pauli Miettinen May 23, 23 Non-Negative Datasets Some datasets are intrinsically non-negative: Counters (e.g., no. occurrences
More informationONLINE NONNEGATIVE MATRIX FACTORIZATION BASED ON KERNEL MACHINES. Fei Zhu, Paul Honeine
ONLINE NONNEGATIVE MATRIX FACTORIZATION BASED ON KERNEL MACINES Fei Zhu, Paul oneine Institut Charles Delaunay (CNRS), Université de Technologie de Troyes, France ABSTRACT Nonnegative matrix factorization
More informationECS289: Scalable Machine Learning
ECS289: Scalable Machine Learning Cho-Jui Hsieh UC Davis Oct 11, 2016 Paper presentations and final project proposal Send me the names of your group member (2 or 3 students) before October 15 (this Friday)
More informationRobust PCA. CS5240 Theoretical Foundations in Multimedia. Leow Wee Kheng
Robust PCA CS5240 Theoretical Foundations in Multimedia Leow Wee Kheng Department of Computer Science School of Computing National University of Singapore Leow Wee Kheng (NUS) Robust PCA 1 / 52 Previously...
More informationA Local Non-Negative Pursuit Method for Intrinsic Manifold Structure Preservation
Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence A Local Non-Negative Pursuit Method for Intrinsic Manifold Structure Preservation Dongdong Chen and Jian Cheng Lv and Zhang Yi
More informationarxiv: v2 [math.oc] 6 Oct 2011
Accelerated Multiplicative Updates and Hierarchical ALS Algorithms for Nonnegative Matrix Factorization Nicolas Gillis 1 and François Glineur 2 arxiv:1107.5194v2 [math.oc] 6 Oct 2011 Abstract Nonnegative
More informationSingle-channel source separation using non-negative matrix factorization
Single-channel source separation using non-negative matrix factorization Mikkel N. Schmidt Technical University of Denmark mns@imm.dtu.dk www.mikkelschmidt.dk DTU Informatics Department of Informatics
More informationPROJECTIVE NON-NEGATIVE MATRIX FACTORIZATION WITH APPLICATIONS TO FACIAL IMAGE PROCESSING
st Reading October 0, 200 8:4 WSPC/-IJPRAI SPI-J068 008 International Journal of Pattern Recognition and Artificial Intelligence Vol. 2, No. 8 (200) 0 c World Scientific Publishing Company PROJECTIVE NON-NEGATIVE
More informationNon-negative matrix factorization and term structure of interest rates
Non-negative matrix factorization and term structure of interest rates Hellinton H. Takada and Julio M. Stern Citation: AIP Conference Proceedings 1641, 369 (2015); doi: 10.1063/1.4906000 View online:
More informationAnalysis of Robust PCA via Local Incoherence
Analysis of Robust PCA via Local Incoherence Huishuai Zhang Department of EECS Syracuse University Syracuse, NY 3244 hzhan23@syr.edu Yi Zhou Department of EECS Syracuse University Syracuse, NY 3244 yzhou35@syr.edu
More informationDimension Reduction Using Nonnegative Matrix Tri-Factorization in Multi-label Classification
250 Int'l Conf. Par. and Dist. Proc. Tech. and Appl. PDPTA'15 Dimension Reduction Using Nonnegative Matrix Tri-Factorization in Multi-label Classification Keigo Kimura, Mineichi Kudo and Lu Sun Graduate
More informationJordan Journal of Mathematics and Statistics (JJMS) 5(3), 2012, pp A NEW ITERATIVE METHOD FOR SOLVING LINEAR SYSTEMS OF EQUATIONS
Jordan Journal of Mathematics and Statistics JJMS) 53), 2012, pp.169-184 A NEW ITERATIVE METHOD FOR SOLVING LINEAR SYSTEMS OF EQUATIONS ADEL H. AL-RABTAH Abstract. The Jacobi and Gauss-Seidel iterative
More informationNon-linear Dimensionality Reduction
Non-linear Dimensionality Reduction CE-725: Statistical Pattern Recognition Sharif University of Technology Spring 2013 Soleymani Outline Introduction Laplacian Eigenmaps Locally Linear Embedding (LLE)
More informationDID: Distributed Incremental Block Coordinate Descent for Nonnegative Matrix Factorization
DID: Distributed Incremental Block Coordinate Descent for Nonnegative Matrix Factorization Tianxiang Gao, Chris Chu Department of Electrical and Computer Engineering, Iowa State University, Ames, IA, 500,
More informationA Unified Approach to Proximal Algorithms using Bregman Distance
A Unified Approach to Proximal Algorithms using Bregman Distance Yi Zhou a,, Yingbin Liang a, Lixin Shen b a Department of Electrical Engineering and Computer Science, Syracuse University b Department
More informationLinear & nonlinear classifiers
Linear & nonlinear classifiers Machine Learning Hamid Beigy Sharif University of Technology Fall 1394 Hamid Beigy (Sharif University of Technology) Linear & nonlinear classifiers Fall 1394 1 / 34 Table
More informationRobust Sparse Recovery via Non-Convex Optimization
Robust Sparse Recovery via Non-Convex Optimization Laming Chen and Yuantao Gu Department of Electronic Engineering, Tsinghua University Homepage: http://gu.ee.tsinghua.edu.cn/ Email: gyt@tsinghua.edu.cn
More informationShiqian Ma, MAT-258A: Numerical Optimization 1. Chapter 9. Alternating Direction Method of Multipliers
Shiqian Ma, MAT-258A: Numerical Optimization 1 Chapter 9 Alternating Direction Method of Multipliers Shiqian Ma, MAT-258A: Numerical Optimization 2 Separable convex optimization a special case is min f(x)
More informationAlgorithm-Hardware Co-Optimization of Memristor-Based Framework for Solving SOCP and Homogeneous QCQP Problems
L.C.Smith College of Engineering and Computer Science Algorithm-Hardware Co-Optimization of Memristor-Based Framework for Solving SOCP and Homogeneous QCQP Problems Ao Ren Sijia Liu Ruizhe Cai Wujie Wen
More informationAutomatic Rank Determination in Projective Nonnegative Matrix Factorization
Automatic Rank Determination in Projective Nonnegative Matrix Factorization Zhirong Yang, Zhanxing Zhu, and Erkki Oja Department of Information and Computer Science Aalto University School of Science and
More informationContraction Methods for Convex Optimization and monotone variational inequalities No.12
XII - 1 Contraction Methods for Convex Optimization and monotone variational inequalities No.12 Linearized alternating direction methods of multipliers for separable convex programming Bingsheng He Department
More informationA Compressive Sensing Based Compressed Neural Network for Sound Source Localization
A Compressive Sensing Based Compressed Neural Network for Sound Source Localization Mehdi Banitalebi Dehkordi Speech Processing Research Lab Yazd University Yazd, Iran mahdi_banitalebi@stu.yazduni.ac.ir
More informationDeep learning / Ian Goodfellow, Yoshua Bengio and Aaron Courville. - Cambridge, MA ; London, Spis treści
Deep learning / Ian Goodfellow, Yoshua Bengio and Aaron Courville. - Cambridge, MA ; London, 2017 Spis treści Website Acknowledgments Notation xiii xv xix 1 Introduction 1 1.1 Who Should Read This Book?
More informationLinear & nonlinear classifiers
Linear & nonlinear classifiers Machine Learning Hamid Beigy Sharif University of Technology Fall 1396 Hamid Beigy (Sharif University of Technology) Linear & nonlinear classifiers Fall 1396 1 / 44 Table
More informationAn Optimization-based Approach to Decentralized Assignability
2016 American Control Conference (ACC) Boston Marriott Copley Place July 6-8, 2016 Boston, MA, USA An Optimization-based Approach to Decentralized Assignability Alborz Alavian and Michael Rotkowitz Abstract
More informationCSE 291. Assignment Spectral clustering versus k-means. Out: Wed May 23 Due: Wed Jun 13
CSE 291. Assignment 3 Out: Wed May 23 Due: Wed Jun 13 3.1 Spectral clustering versus k-means Download the rings data set for this problem from the course web site. The data is stored in MATLAB format as
More informationAdaptive Primal Dual Optimization for Image Processing and Learning
Adaptive Primal Dual Optimization for Image Processing and Learning Tom Goldstein Rice University tag7@rice.edu Ernie Esser University of British Columbia eesser@eos.ubc.ca Richard Baraniuk Rice University
More informationFast Coordinate Descent methods for Non-Negative Matrix Factorization
Fast Coordinate Descent methods for Non-Negative Matrix Factorization Inderjit S. Dhillon University of Texas at Austin SIAM Conference on Applied Linear Algebra Valencia, Spain June 19, 2012 Joint work
More informationDistributed Smooth and Strongly Convex Optimization with Inexact Dual Methods
Distributed Smooth and Strongly Convex Optimization with Inexact Dual Methods Mahyar Fazlyab, Santiago Paternain, Alejandro Ribeiro and Victor M. Preciado Abstract In this paper, we consider a class of
More informationA Fast Augmented Lagrangian Algorithm for Learning Low-Rank Matrices
A Fast Augmented Lagrangian Algorithm for Learning Low-Rank Matrices Ryota Tomioka 1, Taiji Suzuki 1, Masashi Sugiyama 2, Hisashi Kashima 1 1 The University of Tokyo 2 Tokyo Institute of Technology 2010-06-22
More informationHou, Ch. et al. IEEE Transactions on Neural Networks March 2011
Hou, Ch. et al. IEEE Transactions on Neural Networks March 2011 Semi-supervised approach which attempts to incorporate partial information from unlabeled data points Semi-supervised approach which attempts
More informationDual Methods. Lecturer: Ryan Tibshirani Convex Optimization /36-725
Dual Methods Lecturer: Ryan Tibshirani Conve Optimization 10-725/36-725 1 Last time: proimal Newton method Consider the problem min g() + h() where g, h are conve, g is twice differentiable, and h is simple.
More information