References. --- a tentative list of papers to be mentioned in the ICML 2017 tutorial. Recent Advances in Stochastic Convex and Non-Convex Optimization
|
|
|
- Brent Hodges
- 6 years ago
- Views:
Transcription
1 References --- a tentative list of papers to be mentioned in the ICML 2017 tutorial Recent Advances in Stochastic Convex and Non-Convex Optimization Disclaimer: in a quite arbitrary order. 1. [ShalevShwartz-Zhang, 2013]: AccSDCA Accelerated Proximal Stochastic Dual Coordinate Ascent for Regularized Loss Minimization by Shai Shalev-Shwartz, Tong Zhang 2. [Lin-Lu-Xiao, 2014]: APCG An Accelerated Proximal Coordinate Gradient Method and its Application to Regularized Empirical Risk Minimization by Qihang Lin, Zhaosong Lu, Lin Xiao 3. [BenTal-Nemirovski, 2013] Lectures on Modern Convex Optimization By Aharon Ben-Tal and Arkadi Nemirovski 4. [Nemirovski, 2005] Prox-method with rate of convergence o(1/t) for variational inequalities with lipschitz continuous monotone operators and smooth convex-concave saddle point problems. By Arkadi Nemirovski 5. [Chambolle-Pock, 2011] A first-order primal-dual algorithm for convex problems with applications to imaging By Antonin Chambolle and Thomas Pock 6. [LeRoux-Schmidt-Bach, 2012]: SAG A stochastic gradient method with an exponential convergence rate for finite training sets. By Nicolas Le Roux, Mark Schmidt, and Francis R. Bach 7. [Johnson-Zhang, 2013]: SVRG Accelerating stochastic gradient descent using predictive variance reduction. By Rie Johnson and Tong Zhang 8. [Defazio-Bach-LacosteJulien, 2014]: SAGA SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives.
2 By Defazio, A., Bach, F., & Lacoste-Julien, S. 9. [ShalevShwartz-Zhang, 2012]: SDCA Stochastic Dual Coordinate Ascent Methods for Regularized Loss Minimization By Shai Shalev-Shwartz and Tong Zhang 10. [ShalevShwartz-Zhang, 2014]: AccSDCA Accelerated Proximal Stochastic Dual Coordinate As- cent for Regularized Loss Minimization By Shai Shalev-Shwartz and Tong Zhang 11. [Nesterov 2012]: RCDM/ACDM Efficiency of Coordinate Descent Methods on Huge-Scale Optimization Problems. By Yurii Nesterov. 12. [Lee-Sidford, 2013]: ACDM (better proof) Efficient accelerated coordinate descent methods and faster algorithms for solving linear systems. By Yin Tat Lee and Aaron Sidford 13. [AllenZhu-Qu-Richtarik-Yuan, 2016]: NUACDM Even Faster Accelerated Coordinate Descent Using Non-Uniform Sampling By Zeyuan Allen-Zhu, Zheng Qu, Peter Richtarik and Yang Yuan. 14. [Zhang-Xiao, 2015]: SPDC Stochastic Primal-Dual Coordinate Method for Regularized Empirical Risk Minimization By Yuchen Zhang and Lin Xiao. 15. [Lan-Zhou, 2015]: RPDG An optimal randomized incremental gradient method By Guanghui Lan and Yi Zhou. 16. [Frostig-Ge-Kakade-Sidford, 2015]: APPA Un-regularizing: approximate proximal point and faster stochastic algorithms for empirical risk minimization By Roy Frostig, Rong Ge, Sham M. Kakade, and Aaron Sidford 17. [Lin-Mairal-Harchaoui, 2015]: Catalyst A universal catalyst for first-order optimization By Hongzhou Lin, Julien Mairal and Zaid Harchaoui 18. [Zhang-Mahdavi-Jin, 2013] Linear convergence with condition number independent access of full gradients. By Lijun Zhang, Mehrdad Mahdavi, and Rong Jin. 19. [AllenZhu-Yuan, 2015] SVRG++
3 Improved SVRG for Non-Strongly-Convex or Sum-of-Non-Convex Objectives By Zeyuan Allen-Zhu and Yang Yuan 20. [AllenZhu-Hazan 2016b]: reduction Optimal Black-Box Reductions Between Optimization Objectives By Zeyuan Allen-Zhu and Elad Hazan 21. [AllenZhu-Hazan 2016a]: non-convex SVRG Variance Reduction for Faster Non-Convex Optimization By Zeyuan Allen-Zhu and Elad Hazan 22. [Mahdavi-Zhang-Jin 2013] Mixed optimization for smooth functions By Mehrdad Mahdavi, Lijun Zhang, and Rong Jin. 23. [Lei-Jordan, 2017] Less than a Single Pass: Stochastically Controlled Stochastic Gradient By Lihua Lei and Michael I. Jordan. 24. [Lei-Ju-Chen-Jordan, 2017] Nonconvex Finite-Sum Optimization Via SCSG Methods By Lihua Lei, Cheng Ju, Jianbo Chen, and Michael I. Jordan. 25. [Nesterov 2005] Smooth minimization of non-smooth functions By Yurii Nesterov 26. [Su-Boyd-Candes, 2014] A differential equation for modeling nesterovs accelerated gradient method: Theory and insights By Weijie Su, Stephen Boyd, Emmanuel J. Candès 27. [Bubeck-Lee-Singh, 2015] A geometric alternative to Nesterov's accelerated gradient descent By Sébastien Bubeck, Yin Tat Lee, Mohit Singh 28. [Wibisono-Wilson-Jordan, 2016] A Variational Perspective on Accelerated Methods in Optimization By Andre Wibisono, Ashia C. Wilson, Michael I. Jordan 29. [AllenZhu-Orecchia, 2014] Linear Coupling: An Ultimate Unification of Gradient and Mirror Descent By Zeyuan Allen-Zhu and Lorenzo Orecchia. 30. [Nitanda, 2014]
4 Stochastic proximal gradient descent with acceleration techniques. By Atsushi Nitanda. 31. [AllenZhu, 2016] Katyusha: The First Direct Acceleration of Stochastic Gradient Methods By Zeyuan Allen-Zhu 32. [Woodworth-Srebro, 2016] Tight Complexity Bounds for Optimizing Composite Objectives By Blake Woodworth and Nathan Srebro 33. [Frostig-Musco-Musco-Sidford, 2016]: PCR Principal Component Projection Without Principal Component Analysis By Roy Frostig, Cameron Musco, Christopher Musco and Aaron Sidford. 34. [AllenZhu-Li, 2017]: PCR Faster Principal Component Regression and Stable Matrix Chebyshev Approximation By Zeyuan Allen-Zhu and Yuanzhi Li 35. [Shibagaki-Takeuchi, 2017]: mini-batch SPDC Stochastic primal dual coordinate method with nonuniform sampling based on optimality violations By Atsushi Shibagaki and Ichiro Takeuchi. 36. [Murata-Suzuki, 2017]: DASVRDA Doubly accelerated stochastic variance reduced dual averaging method for regularized empirical risk minimization By Tomoya Murata and Taiji Suzuki 37. [Garber et al, 2015]: shift and invert (two papers about the same result) Faster Eigenvector Computation via Shift-and-Invert Preconditioning By Dan Garber, Elad Hazan, Chi Jin, Sham M. Kakade, Cameron Musco, Praneeth Netrapalli, Aaron Sidford [Garber-Hazan, 2015] Fast and Simple PCA via Convex Optimization By Dan Garber and Elad Hazan 38. [AllenZhu-Li, 2016]: LazySVD LazySVD: Even Faster SVD Decomposition Yet Without Agonizing Pain By Zeyuan Allen-Zhu and Yuanzhi Li. 39. [Sun-Luo, 2014] Guaranteed Matrix Completion via Non-convex Factorization By Ruoyu Sun and Zhi-Quan Luo
5 40. [Arora-Ge-Ma-Moitra, 2015] Simple, Efficient, and Neural Algorithms for Sparse Coding By Sanjeev Arora, Rong Ge, Tengyu Ma and Ankur Moitra 41. [Chen-Candès, 2015] Solving Random Quadratic Systems of Equations Is Nearly as Easy as Solving Linear Systems By Yuxin Chen, Emmanuel J. Candès 42. [Nesterov, 2004]: textbook Introductory Lectures on Convex Programming Volume: A Basic course By Yurii Nesterov 43. [Li-Yuan, 2017] Convergence Analysis of Two-layer Neural Networks with ReLU Activation By Yuanzhi Li and Yang Yuan 44. [ShalevShwartz, 2015] SDCA without Duality By Shai Shalev-Shwartz 45. [Reddi et al., 2016a] Stochastic variance reduction for nonconvex optimization By Sashank J Reddi, Ahmed Hefny, Suvrit Sra, Barnabas Poczos, and Alex Smola 46. [Reddi et al., 2016b] Fast incremental method for nonconvex optimization Sashank J Reddi, Suvrit Sra, Barnabás Póczos, and Alex Smola. 47. [Reddi et al., 2016c] Fast Stochastic Methods for Nonsmooth Nonconvex Optimization Sashank J Reddi, Suvrit Sra, Barnabás Póczos, and Alex Smola. 48. [Lei-Ju-Chen-Jordan, 2017] Nonconvex Finite-Sum Optimization Via SCSG Methods By Lihua Lei, Cheng Ju, Jianbo Chen, and Michael I. Jordan 49. [Carmon-Duchi-Hinder-Sidford, 2016] Accelerated Methods for Non-Convex Optimization By Yair Carmon, John C. Duchi, Oliver Hinder and Aaron Sidford 50. [AllenZhu, 2017] Natasha: Faster Non-Convex Stochastic Optimization via Strongly Non-Convex Parameter By Zeyuan Allen-Zhu
6 51. [Carmon- Duchi-Hinder-Sidford, 2017] "Convex Until Proven Guilty": Dimension-Free Acceleration of Gradient Descent on Non-Convex Functions By Yair Carmon, John C. Duchi, Oliver Hinder and Aaron Sidford 52. [Ge-Huang-Jin-Yuan, 2015] Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decomposition By Rong Ge, Furong Huang, Chi Jin, Yang Yuan 53. [Jin et al., 2017] How to Escape Saddle Points Efficiently By Chi Jin, Rong Ge, Praneeth Netrapalli, Sham M. Kakade, Michael I. Jordan 54. [Agarwal et al., 2016] Finding Approximate Local Minima Faster Than Gradient Descent By Naman Agarwal, Zeyuan Allen-Zhu, Brian Bullins, Elad Hazan, Tengyu Ma
arxiv: v1 [math.oc] 18 Mar 2016
![arxiv: v1 [math.oc] 18 Mar 2016](/thumbs/89/98529137.jpg "arxiv: v1 [math.oc] 18 Mar 2016")
Katyusha: Accelerated Variance Reduction for Faster SGD Zeyuan Allen-Zhu zeyuan@csail.mit.edu Princeton University arxiv:1603.05953v1 [math.oc] 18 Mar 016 March 18, 016 Abstract We consider minimizing
Proximal Minimization by Incremental Surrogate Optimization (MISO)
Proximal Minimization by Incremental Surrogate Optimization (MISO) (and a few variants) Julien Mairal Inria, Grenoble ICCOPT, Tokyo, 2016 Julien Mairal, Inria MISO 1/26 Motivation: large-scale machine
Stochastic Variance Reduction for Nonconvex Optimization. Barnabás Póczos
1 Stochastic Variance Reduction for Nonconvex Optimization Barnabás Póczos Contents 2 Stochastic Variance Reduction for Nonconvex Optimization Joint work with Sashank Reddi, Ahmed Hefny, Suvrit Sra, and
Accelerated Randomized Primal-Dual Coordinate Method for Empirical Risk Minimization
Accelerated Randomized Primal-Dual Coordinate Method for Empirical Risk Minimization Lin Xiao (Microsoft Research) Joint work with Qihang Lin (CMU), Zhaosong Lu (Simon Fraser) Yuchen Zhang (UC Berkeley)
Improved Optimization of Finite Sums with Minibatch Stochastic Variance Reduced Proximal Iterations
Improved Optimization of Finite Sums with Miniatch Stochastic Variance Reduced Proximal Iterations Jialei Wang University of Chicago Tong Zhang Tencent AI La Astract jialei@uchicago.edu tongzhang@tongzhang-ml.org
Doubly Accelerated Stochastic Variance Reduced Dual Averaging Method for Regularized Empirical Risk Minimization
Doubly Accelerated Stochastic Variance Reduced Dual Averaging Method for Regularized Empirical Risk Minimization Tomoya Murata Taiji Suzuki More recently, several authors have proposed accelerarxiv:70300439v4
ONLINE VARIANCE-REDUCING OPTIMIZATION
ONLINE VARIANCE-REDUCING OPTIMIZATION Nicolas Le Roux Google Brain nlr@google.com Reza Babanezhad University of British Columbia rezababa@cs.ubc.ca Pierre-Antoine Manzagol Google Brain manzagop@google.com
SVRG++ with Non-uniform Sampling
SVRG++ with Non-uniform Sampling Tamás Kern András György Department of Electrical and Electronic Engineering Imperial College London, London, UK, SW7 2BT {tamas.kern15,a.gyorgy}@imperial.ac.uk Abstract
Optimization for Machine Learning
Optimization for Machine Learning (Lecture 3-A - Convex) SUVRIT SRA Massachusetts Institute of Technology Special thanks: Francis Bach (INRIA, ENS) (for sharing this material, and permitting its use) MPI-IS
A Universal Catalyst for Gradient-Based Optimization
A Universal Catalyst for Gradient-Based Optimization Julien Mairal Inria, Grenoble CIMI workshop, Toulouse, 2015 Julien Mairal, Inria Catalyst 1/58 Collaborators Hongzhou Lin Zaid Harchaoui Publication
Optimization for Machine Learning (Lecture 3-B - Nonconvex)
Optimization for Machine Learning (Lecture 3-B - Nonconvex) SUVRIT SRA Massachusetts Institute of Technology MPI-IS Tübingen Machine Learning Summer School, June 2017 ml.mit.edu Nonconvex problems are
Modern Stochastic Methods. Ryan Tibshirani (notes by Sashank Reddi and Ryan Tibshirani) Convex Optimization
Modern Stochastic Methods Ryan Tibshirani (notes by Sashank Reddi and Ryan Tibshirani) Convex Optimization 10-725 Last time: conditional gradient method For the problem min x f(x) subject to x C where
arxiv: v1 [math.oc] 5 Feb 2018
![arxiv: v1 [math.oc] 5 Feb 2018](/thumbs/82/86108988.jpg "arxiv: v1 [math.oc] 5 Feb 2018")
for Convex-Concave Saddle Point Problems without Strong Convexity Simon S. Du * 1 Wei Hu * arxiv:180.01504v1 math.oc] 5 Feb 018 Abstract We consider the convex-concave saddle point problem x max y fx +
Finite-sum Composition Optimization via Variance Reduced Gradient Descent
Finite-sum Composition Optimization via Variance Reduced Gradient Descent Xiangru Lian Mengdi Wang Ji Liu University of Rochester Princeton University University of Rochester xiangru@yandex.com mengdiw@princeton.edu
Lecture 3: Minimizing Large Sums. Peter Richtárik
Lecture 3: Minimizing Large Sums Peter Richtárik Graduate School in Systems, Op@miza@on, Control and Networks Belgium 2015 Mo@va@on: Machine Learning & Empirical Risk Minimiza@on Training Linear Predictors
Adaptive Negative Curvature Descent with Applications in Non-convex Optimization
Adaptive Negative Curvature Descent with Applications in Non-convex Optimization Mingrui Liu, Zhe Li, Xiaoyu Wang, Jinfeng Yi, Tianbao Yang Department of Computer Science, The University of Iowa, Iowa
arxiv: v3 [math.oc] 23 Jan 2018
![arxiv: v3 [math.oc] 23 Jan 2018](/thumbs/90/104118487.jpg "arxiv: v3 [math.oc] 23 Jan 2018")
Nonconvex Finite-Sum Optimization Via SCSG Methods arxiv:7060956v [mathoc] Jan 08 Lihua Lei UC Bereley lihualei@bereleyedu Cheng Ju UC Bereley cu@bereleyedu Michael I Jordan UC Bereley ordan@statbereleyedu
ON THE INEFFECTIVENESS OF VARIANCE REDUCED OPTIMIZATION FOR DEEP LEARNING
ON THE INEFFECTIVENESS OF VARIANCE REDUCED OPTIMIZATION FOR DEEP LEARNING Anonymous authors Paper under double-blind review ABSTRACT The application of stochastic variance reduction to optimization has
Fast Stochastic Optimization Algorithms for ML
Fast Stochastic Optimization Algorithms for ML Aaditya Ramdas April 20, 205 This lecture is about efficient algorithms for minimizing finite sums min w R d n i= f i (w) or min w R d n f i (w) + λ 2 w 2
Accelerating SVRG via second-order information
Accelerating via second-order information Ritesh Kolte Department of Electrical Engineering rkolte@stanford.edu Murat Erdogdu Department of Statistics erdogdu@stanford.edu Ayfer Özgür Department of Electrical
How to Escape Saddle Points Efficiently? Praneeth Netrapalli Microsoft Research India
How to Escape Saddle Points Efficiently? Praneeth Netrapalli Microsoft Research India Chi Jin UC Berkeley Michael I. Jordan UC Berkeley Rong Ge Duke Univ. Sham M. Kakade U Washington Nonconvex optimization
An Estimate Sequence for Geodesically Convex Optimization
Proceedings of Machine Learning Research vol 75:, 08 An Estimate Sequence for Geodesically Convex Optimization Hongyi Zhang BCS and LIDS, Massachusetts Institute of Technology Suvrit Sra EECS and LIDS,
Tutorial: PART 2. Optimization for Machine Learning. Elad Hazan Princeton University. + help from Sanjeev Arora & Yoram Singer
Tutorial: PART 2 Optimization for Machine Learning Elad Hazan Princeton University + help from Sanjeev Arora & Yoram Singer Agenda 1. Learning as mathematical optimization Stochastic optimization, ERM,
Stochastic optimization: Beyond stochastic gradients and convexity Part I
Stochastic optimization: Beyond stochastic gradients and convexity Part I Francis Bach INRIA - Ecole Normale Supérieure, Paris, France ÉCOLE NORMALE SUPÉRIEURE Joint tutorial with Suvrit Sra, MIT - NIPS
Gradient Descent Can Take Exponential Time to Escape Saddle Points
Gradient Descent Can Take Exponential Time to Escape Saddle Points Simon S. Du Carnegie Mellon University ssdu@cs.cmu.edu Jason D. Lee University of Southern California jasonlee@marshall.usc.edu Barnabás
Limitations on Variance-Reduction and Acceleration Schemes for Finite Sum Optimization
Limitations on Variance-Reduction and Acceleration Schemes for Finite Sum Optimization Yossi Arjevani Department of Computer Science and Applied Mathematics Weizmann Institute of Science Rehovot 7610001,
Optimizing Nonconvex Finite Sums by a Proximal Primal-Dual Method
Optimizing Nonconvex Finite Sums by a Proximal Primal-Dual Method Davood Hajinezhad Iowa State University Davood Hajinezhad Optimizing Nonconvex Finite Sums by a Proximal Primal-Dual Method 1 / 35 Co-Authors
Nesterov s Acceleration
Nesterov s Acceleration Nesterov Accelerated Gradient min X f(x)+ (X) f -smooth. Set s 1 = 1 and = 1. Set y 0. Iterate by increasing t: g t 2 @f(y t ) s t+1 = 1+p 1+4s 2 t 2 y t = x t + s t 1 s t+1 (x
Variance-Reduced and Projection-Free Stochastic Optimization
Elad Hazan Princeton University, Princeton, NJ 08540, USA Haipeng Luo Princeton University, Princeton, NJ 08540, USA EHAZAN@CS.PRINCETON.EDU HAIPENGL@CS.PRINCETON.EDU Abstract The Frank-Wolfe optimization
arxiv: v1 [stat.ml] 20 May 2017
![arxiv: v1 [stat.ml] 20 May 2017](/thumbs/76/73670885.jpg "arxiv: v1 [stat.ml] 20 May 2017")
Stochastic Recursive Gradient Algorithm for Nonconvex Optimization arxiv:1705.0761v1 [stat.ml] 0 May 017 Lam M. Nguyen Industrial and Systems Engineering Lehigh University, USA lamnguyen.mltd@gmail.com
Stochastic Dual Coordinate Ascent Methods for Regularized Loss Minimization
Stochastic Dual Coordinate Ascent Methods for Regularized Loss Minimization Shai Shalev-Shwartz and Tong Zhang School of CS and Engineering, The Hebrew University of Jerusalem Optimization for Machine
Barzilai-Borwein Step Size for Stochastic Gradient Descent
Barzilai-Borwein Step Size for Stochastic Gradient Descent Conghui Tan Shiqian Ma Yu-Hong Dai Yuqiu Qian May 16, 2016 Abstract One of the major issues in stochastic gradient descent (SGD) methods is how
On the Iteration Complexity of Oblivious First-Order Optimization Algorithms
On the Iteration Complexity of Oblivious First-Order Optimization Algorithms Yossi Arjevani Weizmann Institute of Science, Rehovot 7610001, Israel Ohad Shamir Weizmann Institute of Science, Rehovot 7610001,
Minimizing Finite Sums with the Stochastic Average Gradient Algorithm
Minimizing Finite Sums with the Stochastic Average Gradient Algorithm Joint work with Nicolas Le Roux and Francis Bach University of British Columbia Context: Machine Learning for Big Data Large-scale
arxiv: v2 [math.oc] 1 Nov 2017
![arxiv: v2 [math.oc] 1 Nov 2017](/thumbs/86/93325763.jpg "arxiv: v2 [math.oc] 1 Nov 2017")
Stochastic Non-convex Optimization with Strong High Probability Second-order Convergence arxiv:1710.09447v [math.oc] 1 Nov 017 Mingrui Liu, Tianbao Yang Department of Computer Science The University of
SEGA: Variance Reduction via Gradient Sketching
SEGA: Variance Reduction via Gradient Sketching Filip Hanzely 1 Konstantin Mishchenko 1 Peter Richtárik 1,2,3 1 King Abdullah University of Science and Technology, 2 University of Edinburgh, 3 Moscow Institute
arxiv: v2 [cs.lg] 14 Sep 2017
![arxiv: v2 [cs.lg] 14 Sep 2017](/thumbs/77/76351108.jpg "arxiv: v2 [cs.lg] 14 Sep 2017")
Elad Hazan Princeton University, Princeton, NJ 08540, USA Haipeng Luo Princeton University, Princeton, NJ 08540, USA EHAZAN@CS.PRINCETON.EDU HAIPENGL@CS.PRINCETON.EDU arxiv:160.0101v [cs.lg] 14 Sep 017
Stochastic Gradient Descent with Variance Reduction
Stochastic Gradient Descent with Variance Reduction Rie Johnson, Tong Zhang Presenter: Jiawen Yao March 17, 2015 Rie Johnson, Tong Zhang Presenter: JiawenStochastic Yao Gradient Descent with Variance Reduction
arxiv: v1 [math.oc] 7 Jun 2018
![arxiv: v1 [math.oc] 7 Jun 2018](/thumbs/89/99380826.jpg "arxiv: v1 [math.oc] 7 Jun 2018")
Towards Riemannian Accelerated Gradient Methods Hongyi Zhang BCS & LIDS, MIT hongyiz@mit.edu Suvrit Sra EECS & LIDS, MIT suvrit@mit.edu arxiv:806.08v [math.oc] 7 Jun 08 Abstract We propose a Riemannian
Exploiting Strong Convexity from Data with Primal-Dual First-Order Algorithms
Exploiting Strong Convexit from Data with Primal-Dual First-Order Algorithms Jialei Wang Lin Xiao Abstract We consider empirical ris minimization of linear predictors with convex loss functions. Such problems
Big Data Analytics: Optimization and Randomization
Big Data Analytics: Optimization and Randomization Tianbao Yang Tutorial@ACML 2015 Hong Kong Department of Computer Science, The University of Iowa, IA, USA Nov. 20, 2015 Yang Tutorial for ACML 15 Nov.
Asynchronous Mini-Batch Gradient Descent with Variance Reduction for Non-Convex Optimization
Proceedings of the hirty-first AAAI Conference on Artificial Intelligence (AAAI-7) Asynchronous Mini-Batch Gradient Descent with Variance Reduction for Non-Convex Optimization Zhouyuan Huo Dept. of Computer
Provable Non-Convex Min-Max Optimization
Provable Non-Convex Min-Max Optimization Mingrui Liu, Hassan Rafique, Qihang Lin, Tianbao Yang Department of Computer Science, The University of Iowa, Iowa City, IA, 52242 Department of Mathematics, The
COR-OPT Seminar Reading List Sp 18
COR-OPT Seminar Reading List Sp 18 Damek Davis January 28, 2018 References [1] S. Tu, R. Boczar, M. Simchowitz, M. Soltanolkotabi, and B. Recht. Low-rank Solutions of Linear Matrix Equations via Procrustes
Katyusha: The First Direct Acceleration of Stochastic Gradient Methods
Journal of Machine Learning Research 8 8-5 Submitted 8/6; Revised 5/7; Published 6/8 : The First Direct Acceleration of Stochastic Gradient Methods Zeyuan Allen-Zhu Microsoft Research AI Redmond, WA 985,
First Efficient Convergence for Streaming k-pca: a Global, Gap-Free, and Near-Optimal Rate
58th Annual IEEE Symposium on Foundations of Computer Science First Efficient Convergence for Streaming k-pca: a Global, Gap-Free, and Near-Optimal Rate Zeyuan Allen-Zhu Microsoft Research zeyuan@csail.mit.edu
On Gradient-Based Optimization: Accelerated, Stochastic, Asynchronous, Distributed
On Gradient-Based Optimization: Accelerated, Stochastic, Asynchronous, Distributed Michael I. Jordan University of California, Berkeley February 9, 2017 Modern Optimization and Large-Scale Data Analysis
Communication-efficient Algorithms for Distributed Stochastic Principal Component Analysis
Communication-efficient Algorithms for Distributed Stochastic Principal Component Analysis Dan Garber Ohad Shamir Nathan Srebro 3 Abstract We study the fundamental problem of Principal Component Analysis
Incremental Reshaped Wirtinger Flow and Its Connection to Kaczmarz Method
Incremental Reshaped Wirtinger Flow and Its Connection to Kaczmarz Method Huishuai Zhang Department of EECS Syracuse University Syracuse, NY 3244 hzhan23@syr.edu Yingbin Liang Department of EECS Syracuse
Composite nonlinear models at scale
Composite nonlinear models at scale Dmitriy Drusvyatskiy Mathematics, University of Washington Joint work with D. Davis (Cornell), M. Fazel (UW), A.S. Lewis (Cornell) C. Paquette (Lehigh), and S. Roy (UW)
ON THE INSUFFICIENCY OF EXISTING MOMENTUM
ON THE INSUFFICIENCY OF EXISTING MOMENTUM SCHEMES FOR STOCHASTIC OPTIMIZATION Rahul Kidambi 1, Praneeth Netrapalli 2, Prateek Jain 2 and Sham M. Kakade 1 1 University of Washington Seattle 2 Microsoft
Stochastic Dual Coordinate Ascent with Adaptive Probabilities
Dominik Csiba Zheng Qu Peter Richtárik University of Edinburgh CDOMINIK@GMAIL.COM ZHENG.QU@ED.AC.UK PETER.RICHTARIK@ED.AC.UK Abstract This paper introduces AdaSDCA: an adaptive variant of stochastic dual
Stochastic and online algorithms
Stochastic and online algorithms stochastic gradient method online optimization and dual averaging method minimizing finite average Stochastic and online optimization 6 1 Stochastic optimization problem
Accelerated Stochastic Block Coordinate Gradient Descent for Sparsity Constrained Nonconvex Optimization
Accelerated Stochastic Block Coordinate Gradient Descent for Sparsity Constrained Nonconvex Optimization Jinghui Chen Department of Systems and Information Engineering University of Virginia Quanquan Gu
A Second-Order Path-Following Algorithm for Unconstrained Convex Optimization
A Second-Order Path-Following Algorithm for Unconstrained Convex Optimization Yinyu Ye Department is Management Science & Engineering and Institute of Computational & Mathematical Engineering Stanford
Stochastic Optimization with Variance Reduction for Infinite Datasets with Finite Sum Structure
Stochastic Optimization with Variance Reduction for Infinite Datasets with Finite Sum Structure Alberto Bietti Julien Mairal Inria Grenoble (Thoth) March 21, 2017 Alberto Bietti Stochastic MISO March 21,
Large-scale Stochastic Optimization
Large-scale Stochastic Optimization 11-741/641/441 (Spring 2016) Hanxiao Liu hanxiaol@cs.cmu.edu March 24, 2016 1 / 22 Outline 1. Gradient Descent (GD) 2. Stochastic Gradient Descent (SGD) Formulation
Incremental and Stochastic Majorization-Minimization Algorithms for Large-Scale Machine Learning
Incremental and Stochastic Majorization-Minimization Algorithms for Large-Scale Machine Learning Julien Mairal Inria, LEAR Team, Grenoble Journées MAS, Toulouse Julien Mairal Incremental and Stochastic
On the Local Minima of the Empirical Risk
On the Local Minima of the Empirical Risk Chi Jin University of California, Berkeley chijin@cs.berkeley.edu Rong Ge Duke University rongge@cs.duke.edu Lydia T. Liu University of California, Berkeley lydiatliu@cs.berkeley.edu
Trade-Offs in Distributed Learning and Optimization
Trade-Offs in Distributed Learning and Optimization Ohad Shamir Weizmann Institute of Science Includes joint works with Yossi Arjevani, Nathan Srebro and Tong Zhang IHES Workshop March 2016 Distributed
Stochastic Cubic Regularization for Fast Nonconvex Optimization
Stochastic Cubic Regularization for Fast Nonconvex Optimization Nilesh Tripuraneni Mitchell Stern Chi Jin Jeffrey Regier Michael I. Jordan {nilesh tripuraneni,mitchell,chijin,regier}@berkeley.edu jordan@cs.berkeley.edu
Contents. 1 Introduction. 1.1 History of Optimization ALG-ML SEMINAR LISSA: LINEAR TIME SECOND-ORDER STOCHASTIC ALGORITHM FEBRUARY 23, 2016
ALG-ML SEMINAR LISSA: LINEAR TIME SECOND-ORDER STOCHASTIC ALGORITHM FEBRUARY 23, 2016 LECTURERS: NAMAN AGARWAL AND BRIAN BULLINS SCRIBE: KIRAN VODRAHALLI Contents 1 Introduction 1 1.1 History of Optimization.....................................
Third-order Smoothness Helps: Even Faster Stochastic Optimization Algorithms for Finding Local Minima
Third-order Smoothness elps: Even Faster Stochastic Optimization Algorithms for Finding Local Minima Yaodong Yu and Pan Xu and Quanquan Gu arxiv:171.06585v1 [math.oc] 18 Dec 017 Abstract We propose stochastic
arxiv: v2 [math.oc] 5 Nov 2017
![arxiv: v2 [math.oc] 5 Nov 2017](/thumbs/87/96509293.jpg "arxiv: v2 [math.oc] 5 Nov 2017")
Gradient Descent Can Take Exponential Time to Escape Saddle Points arxiv:175.1412v2 [math.oc] 5 Nov 217 Simon S. Du Carnegie Mellon University ssdu@cs.cmu.edu Jason D. Lee University of Southern California
Solving Ridge Regression using Sketched Preconditioned SVRG
Alon Gonen The Hebrew University Francesco Orabona Yahoo Research, 9 West 43rd Street, 0036 New York, NY, USA Shai Shalev-Shwartz The Hebrew University ALONGNN@CS.HUJI.AC.IL FRANCESCO@ORABONA.COM SHAIS@CS.HUJI.AC.IL
A Unified Analysis of Stochastic Momentum Methods for Deep Learning
A Unified Analysis of Stochastic Momentum Methods for Deep Learning Yan Yan,2, Tianbao Yang 3, Zhe Li 3, Qihang Lin 4, Yi Yang,2 SUSTech-UTS Joint Centre of CIS, Southern University of Science and Technology
Stochastic Optimization
Introduction Related Work SGD Epoch-GD LM A DA NANJING UNIVERSITY Lijun Zhang Nanjing University, China May 26, 2017 Introduction Related Work SGD Epoch-GD Outline 1 Introduction 2 Related Work 3 Stochastic
Coordinate Descent Faceoff: Primal or Dual?
JMLR: Workshop and Conference Proceedings 83:1 22, 2018 Algorithmic Learning Theory 2018 Coordinate Descent Faceoff: Primal or Dual? Dominik Csiba peter.richtarik@ed.ac.uk School of Mathematics University
SDNA: Stochastic Dual Newton Ascent for Empirical Risk Minimization
Zheng Qu ZHENGQU@HKU.HK Department of Mathematics, The University of Hong Kong, Hong Kong Peter Richtárik PETER.RICHTARIK@ED.AC.UK School of Mathematics, The University of Edinburgh, UK Martin Takáč TAKAC.MT@GMAIL.COM
arxiv: v1 [math.oc] 9 Oct 2018
![arxiv: v1 [math.oc] 9 Oct 2018](/thumbs/90/103686186.jpg "arxiv: v1 [math.oc] 9 Oct 2018")
Cubic Regularization with Momentum for Nonconvex Optimization Zhe Wang Yi Zhou Yingbin Liang Guanghui Lan Ohio State University Ohio State University zhou.117@osu.edu liang.889@osu.edu Ohio State University
arxiv: v2 [stat.ml] 16 Jun 2015
![arxiv: v2 [stat.ml] 16 Jun 2015](/thumbs/89/97564791.jpg "arxiv: v2 [stat.ml] 16 Jun 2015")
Semi-Stochastic Gradient Descent Methods Jakub Konečný Peter Richtárik arxiv:1312.1666v2 [stat.ml] 16 Jun 2015 School of Mathematics University of Edinburgh United Kingdom June 15, 2015 (first version:
STA141C: Big Data & High Performance Statistical Computing
STA141C: Big Data & High Performance Statistical Computing Lecture 8: Optimization Cho-Jui Hsieh UC Davis May 9, 2017 Optimization Numerical Optimization Numerical Optimization: min X f (X ) Can be applied
Randomized Coordinate Descent with Arbitrary Sampling: Algorithms and Complexity
Randomized Coordinate Descent with Arbitrary Sampling: Algorithms and Complexity Zheng Qu University of Hong Kong CAM, 23-26 Aug 2016 Hong Kong based on joint work with Peter Richtarik and Dominique Cisba(University
Stochastic gradient methods for machine learning
Stochastic gradient methods for machine learning Francis Bach INRIA - Ecole Normale Supérieure, Paris, France Joint work with Eric Moulines, Nicolas Le Roux and Mark Schmidt - January 2013 Context Machine
Accelerating Stochastic Gradient Descent for Least Squares Regression
Proceedings of Machine Learning Research vol 75: 6, 8 3st Annual Conference on Learning Theory Accelerating Stochastic Gradient Descent for Least Squares Regression Prateek Jain Praneeth Netrapalli Microsoft
A Conservation Law Method in Optimization
A Conservation Law Method in Optimization Bin Shi Florida International University Tao Li Florida International University Sundaraja S. Iyengar Florida International University Abstract bshi1@cs.fiu.edu
arxiv: v2 [stat.ml] 3 Jun 2017
![arxiv: v2 [stat.ml] 3 Jun 2017](/thumbs/71/65772134.jpg "arxiv: v2 [stat.ml] 3 Jun 2017")
: A Novel Method for Machine Learning Problems Using Stochastic Recursive Gradient Lam M. Nguyen Jie Liu Katya Scheinberg Martin Takáč arxiv:703.000v [stat.ml] 3 Jun 07 Abstract In this paper, we propose
Course Notes for EE227C (Spring 2018): Convex Optimization and Approximation
Course Notes for EE227C (Spring 2018): Convex Optimization and Approximation Instructor: Moritz Hardt Email: hardt+ee227c@berkeley.edu Graduate Instructor: Max Simchowitz Email: msimchow+ee227c@berkeley.edu
Stochastic Convex Optimization: Faster Local Growth Implies Faster Global Convergence
Stochastic Convex Optimization: Faster Local Growth Implies Faster Global Convergence Yi Xu 1 Qihang Lin 2 Tianbao Yang 1 Abstract In this paper, a new theory is developed for firstorder stochastic convex
A Greedy Framework for First-Order Optimization
A Greedy Framework for First-Order Optimization Jacob Steinhardt Department of Computer Science Stanford University Stanford, CA 94305 jsteinhardt@cs.stanford.edu Jonathan Huggins Department of EECS Massachusetts
On Computational Thinking, Inferential Thinking and Data Science
On Computational Thinking, Inferential Thinking and Data Science Michael I. Jordan University of California, Berkeley December 17, 2016 A Job Description, circa 2015 Your Boss: I need a Big Data system
A Stochastic PCA Algorithm with an Exponential Convergence Rate. Ohad Shamir
A Stochastic PCA Algorithm with an Exponential Convergence Rate Ohad Shamir Weizmann Institute of Science NIPS Optimization Workshop December 2014 Ohad Shamir Stochastic PCA with Exponential Convergence
A Brief Overview of Practical Optimization Algorithms in the Context of Relaxation
A Brief Overview of Practical Optimization Algorithms in the Context of Relaxation Zhouchen Lin Peking University April 22, 2018 Too Many Opt. Problems! Too Many Opt. Algorithms! Zero-th order algorithms:
Second-Order Stochastic Optimization for Machine Learning in Linear Time
Journal of Machine Learning Research 8 (207) -40 Submitted 9/6; Revised 8/7; Published /7 Second-Order Stochastic Optimization for Machine Learning in Linear Time Naman Agarwal Computer Science Department
Barzilai-Borwein Step Size for Stochastic Gradient Descent
Barzilai-Borwein Step Size for Stochastic Gradient Descent Conghui Tan The Chinese University of Hong Kong chtan@se.cuhk.edu.hk Shiqian Ma The Chinese University of Hong Kong sqma@se.cuhk.edu.hk Yu-Hong
ECS289: Scalable Machine Learning
ECS289: Scalable Machine Learning Cho-Jui Hsieh UC Davis Sept 29, 2016 Outline Convex vs Nonconvex Functions Coordinate Descent Gradient Descent Newton s method Stochastic Gradient Descent Numerical Optimization
Adaptive restart of accelerated gradient methods under local quadratic growth condition
Adaptive restart of accelerated gradient methods under local quadratic growth condition Olivier Fercoq Zheng Qu September 6, 08 arxiv:709.0300v [math.oc] 7 Sep 07 Abstract By analyzing accelerated proximal
A DELAYED PROXIMAL GRADIENT METHOD WITH LINEAR CONVERGENCE RATE. Hamid Reza Feyzmahdavian, Arda Aytekin, and Mikael Johansson
204 IEEE INTERNATIONAL WORKSHOP ON ACHINE LEARNING FOR SIGNAL PROCESSING, SEPT. 2 24, 204, REIS, FRANCE A DELAYED PROXIAL GRADIENT ETHOD WITH LINEAR CONVERGENCE RATE Hamid Reza Feyzmahdavian, Arda Aytekin,
arxiv: v3 [math.oc] 1 Mar 2018
![arxiv: v3 [math.oc] 1 Mar 2018](/thumbs/94/120092919.jpg "arxiv: v3 [math.oc] 1 Mar 2018")
1 40 First-order Stochastic Algorithms for Escaping From Saddle Points in Almost Linear Time arxiv:1711.01944v3 [math.oc] 1 Mar 2018 Yi Xu yi-xu@uiowa.edu Rong Jin jinrong.jr@ailibaba-inc.com Tianbao Yang
Tight Complexity Bounds for Optimizing Composite Objectives
Tight Coplexity Bounds for Optiizing Coposite Objectives Blake Woodworth Toyota Technological Institute at Chicago Chicago, IL, 60637 blake@ttic.edu Nathan Srebro Toyota Technological Institute at Chicago
Stochastic gradient methods for machine learning
Stochastic gradient methods for machine learning Francis Bach INRIA - Ecole Normale Supérieure, Paris, France Joint work with Eric Moulines, Nicolas Le Roux and Mark Schmidt - September 2012 Context Machine
Convergence of Cubic Regularization for Nonconvex Optimization under KŁ Property
Convergence of Cubic Regularization for Nonconvex Optimization under KŁ Property Yi Zhou Department of ECE The Ohio State University zhou.1172@osu.edu Zhe Wang Department of ECE The Ohio State University
Journal Club. A Universal Catalyst for First-Order Optimization (H. Lin, J. Mairal and Z. Harchaoui) March 8th, CMAP, Ecole Polytechnique 1/19
Journal Club A Universal Catalyst for First-Order Optimization (H. Lin, J. Mairal and Z. Harchaoui) CMAP, Ecole Polytechnique March 8th, 2018 1/19 Plan 1 Motivations 2 Existing Acceleration Methods 3 Universal
Pavel Dvurechensky Alexander Gasnikov Alexander Tiurin. July 26, 2017
Randomized Similar Triangles Method: A Unifying Framework for Accelerated Randomized Optimization Methods Coordinate Descent, Directional Search, Derivative-Free Method) Pavel Dvurechensky Alexander Gasnikov
Linear Convergence under the Polyak-Łojasiewicz Inequality
Linear Convergence under the Polyak-Łojasiewicz Inequality Hamed Karimi, Julie Nutini and Mark Schmidt The University of British Columbia LCI Forum February 28 th, 2017 1 / 17 Linear Convergence of Gradient-Based
Stochastic Gradient Descent with Only One Projection
Stochastic Gradient Descent with Only One Projection Mehrdad Mahdavi, ianbao Yang, Rong Jin, Shenghuo Zhu, and Jinfeng Yi Dept. of Computer Science and Engineering, Michigan State University, MI, USA Machine
The proximal point method revisited
The proimal point method revisited Dmitriy Drusvyatskiy Abstract In this short survey, I revisit the role of the proimal point method in large scale optimization. I focus on three recent eamples: a proimally
NESTT: A Nonconvex Primal-Dual Splitting Method for Distributed and Stochastic Optimization
ESTT: A onconvex Primal-Dual Splitting Method for Distributed and Stochastic Optimization Davood Hajinezhad, Mingyi Hong Tuo Zhao Zhaoran Wang Abstract We study a stochastic and distributed algorithm for
arxiv: v1 [cs.lg] 10 Jan 2019
![arxiv: v1 [cs.lg] 10 Jan 2019](/thumbs/93/113066689.jpg "arxiv: v1 [cs.lg] 10 Jan 2019")
Quantized Epoch-SGD for Communication-Efficient Distributed Learning arxiv:1901.0300v1 [cs.lg] 10 Jan 2019 Shen-Yi Zhao Hao Gao Wu-Jun Li Department of Computer Science and Technology Nanjing University,
Approximate Second Order Algorithms. Seo Taek Kong, Nithin Tangellamudi, Zhikai Guo
Approximate Second Order Algorithms Seo Taek Kong, Nithin Tangellamudi, Zhikai Guo Why Second Order Algorithms? Invariant under affine transformations e.g. stretching a function preserves the convergence