On stochas*c gradient descent, flatness and generaliza*on Yoshua Bengio
|
|
- Catherine Welch
- 5 years ago
- Views:
Transcription
1 On stochas*c gradient descent, flatness and generaliza*on Yoshua Bengio July 14, 2018 ICML 2018 Workshop on nonconvex op=miza=on
2 Disentangling optimization and generalization The tradi=onal ML picture is that op=miza=on and generaliza=on are neatly separated aspects That makes theory easier to handle, separately Unfortunately not the case SGD variants influence op=miza=on AND generaliza=on
3 Memorization in Deep Networks Mostly from preprint arxiv: Devansh Arpit, Stanisław Jastrzębski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, Simon Lacoste-Julien
4 Memorization in Deep Networks Deep networks trained with SGD generalize well due to its implicit regulariza=on effect (Zhang et al 2016) Deep networks achieve ~100% train accuracy on random data (Zhang et al 2016) Do deep networks also memorize real data?
5 Real data has Dominant Patterns Real data: some samples are learned first. Random data: samples are learned in arbitrary order. Frac=on of =mes each of 1000 samples is classified correctly ajer 1 epoch across 100 runs
6 Larger Margin on Real data Real data: distance from decision boundary is large Cri=cal sample ra=o = frac=on of samples which have adversarial examples in their vicinity Random data: distance from decision boundary is small
7 Patterns come First Valida=on accuracy peaks before falling Panerns in real data learned before overfiong noise Train (full) and valida=on (doned) accuracy on MNIST during training with noisy labels
8 Regularization Hinders Memorization Dropout is best at hindering memoriza=on Maintains performance on real data for reduced memoriza=on on random data. Best valida=on performance (picked across hyper parameter grid) on real data vs. training performance on noise labels for the same model, for different regularizers.
9 Take Home Message DNNs learn pa:erns before memorizing noise Why? Does it have to do with SGD?
10 On the relevance of loss function geometry for generalization Laurent Dinh, Razvan Pascanu, Samy Bengio, Yoshua Bengio
11 Flatness
12 Reparametrization Differentiation at critical point Flat minima " Sharp minima Sharp minima" Flat minima
13 Reparametrization Sharp minima can generalize Flat minima can poorly generalize
14 " Eppur, si muove!" And yet, it generalizes!
15 Factors influencing Minima in SGD Mostly from preprint arxiv: Stanisław Jastrzębski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, Amos Storkey
16 Behavior of SGD Small mini-batch finds wider minima (Keskar et al 2016) What dynamics/factors govern the quality of minima found by SGD?
17 <latexit sha1_base64="njgxlgzd+msjarm0lhehhvkd0vc=">aaaelhicjvnnb9naej3ufjtw0rsoxcxsphigsiokqfwquhdgucrck9vvtxbxirxba9nrqmx5x/al+blc4cabia4i/gvvppuipkluo8sz7817o7o2wyjnkjscfuusejcux7m6eq17/cbnw2u99dtvk1oxkr5hjjxlxqgqnsa5htvepnqvklxkwltvhrmxzo8e67jktp7gnht6ifotpimtsflah71nqvyqqakkvdpepf6rzsbtdhrgtwcn2qr2qbsgl5j2sncfdozy+eetkuv65k4ds97jkkajmhrrtrlpyskit0lrhdineq2pahzadbaswsk8ppa60nao0qhqqgf4n2c179aca/asrb1hlxs/ekqf7knjufci59184wtxzvrf3o14cm8nuifokwnqaqr0f7p55uv1piulmj7idalmkgth6slnusupcof+x1nzobtaod8cxykprdk/z180lczoz6ue/ywvjpi6cru1/zyuuc8c2ds5r0wmyohfaof9cqgnvxd3fnigxti7n5xdu0gof+7czqnvhxfaesosbznkzj3nkbwommkl0qnwrtaxnmag9rebqcj5hr6l4ew/pyzpyzz8zrjftlts2qkpwgyxudxcaobrxdcwwn73zyfjrchtwej1o/72c/dzrnjdukebepuf0za9pb0ay/sp9ik+0xfvo/fv++79oc1d6tjnhtpzet//aphj44m=</latexit> <latexit sha1_base64="njgxlgzd+msjarm0lhehhvkd0vc=">aaaelhicjvnnb9naej3ufjtw0rsoxcxsphigsiokqfwquhdgucrck9vvtxbxirxba9nrqmx5x/al+blc4cabia4i/gvvppuipkluo8sz7817o7o2wyjnkjscfuusejcux7m6eq17/cbnw2u99dtvk1oxkr5hjjxlxqgqnsa5htvepnqvklxkwltvhrmxzo8e67jktp7gnht6ifotpimtsflah71nqvyqqakkvdpepf6rzsbtdhrgtwcn2qr2qbsgl5j2sncfdozy+eetkuv65k4ds97jkkajmhrrtrlpyskit0lrhdineq2pahzadbaswsk8ppa60nao0qhqqgf4n2c179aca/asrb1hlxs/ekqf7knjufci59184wtxzvrf3o14cm8nuifokwnqaqr0f7p55uv1piulmj7idalmkgth6slnusupcof+x1nzobtaod8cxykprdk/z180lczoz6ue/ywvjpi6cru1/zyuuc8c2ds5r0wmyohfaof9cqgnvxd3fnigxti7n5xdu0gof+7czqnvhxfaesosbznkzj3nkbwommkl0qnwrtaxnmag9rebqcj5hr6l4ew/pyzpyzz8zrjftlts2qkpwgyxudxcaobrxdcwwn73zyfjrchtwej1o/72c/dzrnjdukebepuf0za9pb0ay/sp9ik+0xfvo/fv++79oc1d6tjnhtpzet//aphj44m=</latexit> <latexit sha1_base64="njgxlgzd+msjarm0lhehhvkd0vc=">aaaelhicjvnnb9naej3ufjtw0rsoxcxsphigsiokqfwquhdgucrck9vvtxbxirxba9nrqmx5x/al+blc4cabia4i/gvvppuipkluo8sz7817o7o2wyjnkjscfuusejcux7m6eq17/cbnw2u99dtvk1oxkr5hjjxlxqgqnsa5htvepnqvklxkwltvhrmxzo8e67jktp7gnht6ifotpimtsflah71nqvyqqakkvdpepf6rzsbtdhrgtwcn2qr2qbsgl5j2sncfdozy+eetkuv65k4ds97jkkajmhrrtrlpyskit0lrhdineq2pahzadbaswsk8ppa60nao0qhqqgf4n2c179aca/asrb1hlxs/ekqf7knjufci59184wtxzvrf3o14cm8nuifokwnqaqr0f7p55uv1piulmj7idalmkgth6slnusupcof+x1nzobtaod8cxykprdk/z180lczoz6ue/ywvjpi6cru1/zyuuc8c2ds5r0wmyohfaof9cqgnvxd3fnigxti7n5xdu0gof+7czqnvhxfaesosbznkzj3nkbwommkl0qnwrtaxnmag9rebqcj5hr6l4ew/pyzpyzz8zrjftlts2qkpwgyxudxcaobrxdcwwn73zyfjrchtwej1o/72c/dzrnjdukebepuf0za9pb0ay/sp9ik+0xfvo/fv++79oc1d6tjnhtpzet//aphj44m=</latexit> <latexit sha1_base64="njgxlgzd+msjarm0lhehhvkd0vc=">aaaelhicjvnnb9naej3ufjtw0rsoxcxsphigsiokqfwquhdgucrck9vvtxbxirxba9nrqmx5x/al+blc4cabia4i/gvvppuipkluo8sz7817o7o2wyjnkjscfuusejcux7m6eq17/cbnw2u99dtvk1oxkr5hjjxlxqgqnsa5htvepnqvklxkwltvhrmxzo8e67jktp7gnht6ifotpimtsflah71nqvyqqakkvdpepf6rzsbtdhrgtwcn2qr2qbsgl5j2sncfdozy+eetkuv65k4ds97jkkajmhrrtrlpyskit0lrhdineq2pahzadbaswsk8ppa60nao0qhqqgf4n2c179aca/asrb1hlxs/ekqf7knjufci59184wtxzvrf3o14cm8nuifokwnqaqr0f7p55uv1piulmj7idalmkgth6slnusupcof+x1nzobtaod8cxykprdk/z180lczoz6ue/ywvjpi6cru1/zyuuc8c2ds5r0wmyohfaof9cqgnvxd3fnigxti7n5xdu0gof+7czqnvhxfaesosbznkzj3nkbwommkl0qnwrtaxnmag9rebqcj5hr6l4ew/pyzpyzz8zrjftlts2qkpwgyxudxcaobrxdcwwn73zyfjrchtwej1o/72c/dzrnjdukebepuf0za9pb0ay/sp9ik+0xfvo/fv++79oc1d6tjnhtpzet//aphj44m=</latexit> SGD as Stochastic Differential Equation Mini-batch gradient g (S) (θ) (due to CLT), batch size S: SGD with learning rate η is described by: Con=nuous stochas=c differen=al equa=on (SDE) form: (Li et al If small enough learning rate, ie. small steps Note: C(θ) = B(θ) T B(θ)
18 Equilibrium Distribution of SGD The equilibrium distribu=on of this SDE is given by: ~Inverse rela=on between loss and density Noise n controls the granularity of the equilibrium distribu=on n = η/s Note: η = learning rate, S = batch size, σ 2 = fixed isotropic gradient variance
19 SGD Moves a Cloud of Points Consider the last k values of θ Form a cloud of points The cloud gradually moves with SGD updates The width of the cloud grows with the noise level (l.rate/bs) It cannot go in valleys sharper than that width 19
20 Implications of the Theory Probability of ending in a minima A described by Hessian H A : In general, minima with larger volume is favored more (simply because it has higher probability mass) Higher noise n priori=zes width (volume) over depth Final equilibrium distribu=on is unchanged when learning rate and batch size are scaled propor=onally η βη, S βs Note: n = η/s, η = learning rate, S = batch size, σ 2 = fixed isotropic gradient variance
21 Experimental Results Smaller Noise Sharper Bowl Equal noise Equal Width
22 Same Noise - Same Learning Dynamics Theory talks about final equilibrium distribu=on but seems to apply along trajectory as well But even learning dynamics is similar when learning rate and batch size are scaled propor=onally η βη, S βs Cyclic Learning Rate and Cyclic Batch-size Constant Learning Rate and Constant Batch-size
23 Take Home Messages DNNs learn pa:erns before memorizing noise Regulariza*on hinders memoriza*on The quality of final minima and learning dynamics is similar when learning rate and batch size are scaled propor*onally Larger noise favors large volume minima over deep ones Larger noise (e.g. due to BS or l.rate) hinders memoriza*on
24 A Walk with SGD Xing, Arpit, Tsirigotis & Bengio ArXiv: Interpolate in parameter space between minibatch SGD updates and see convex shape Ajer ini=al phase, updates bounce off valley floor, which monotonically improves, traversing larger distances with smaller batch sizes (BS) Learning rate: height from floor BS: explora=on noise Pure GD gets stuck on floor, while SGD finds flaner regions, 24 which generalize bener
25 Sharpest Directions Along the SGD Trajectory (Jastrzębski, Kenton, Ballas, Fischer, Bengio, Storkey) Even at the beginning of training, a high learning rate or small batch size influences SGD to visit flaner loss regions. the largest eigenvalues appears to always follow a similar panern, with a fast increase in the early phase and a decrease thereajer, where the peak value is determined by the learning rate and batch size. altering the learning rate just in the direc=on of the eigenvectors associated with the largest eigenvalues, SGD can be steered towards regions which are an order of magnitude sharper but correspond to models with similar generaliza=on, confirming that curvature of the endpoint found by SGD is not predic*ve of its generaliza*on proper*es. 25
26 Montreal Ins*tute for Learning Algorithms
arxiv: v1 [cs.lg] 7 Jan 2019
Generalization in Deep Networks: The Role of Distance from Initialization arxiv:1901672v1 [cs.lg] 7 Jan 2019 Vaishnavh Nagarajan Computer Science Department Carnegie-Mellon University Pittsburgh, PA 15213
More informationA BAYESIAN PERSPECTIVE ON GENERALIZATION AND STOCHASTIC GRADIENT DESCENT
A BAYESIAN PERSPECTIVE ON GENERALIZATION AND STOCHASTIC GRADIENT DESCENT Samuel L. Smith & Quoc V. Le Google Brain {slsmith, qvl}@google.com ABSTRACT We consider two questions at the heart of machine learning;
More informationarxiv: v2 [cs.lg] 15 Jun 2018
Three Factors Influencing Minima in SGD Stanislaw Jastrzębski 123, Zachary Kenton 13, Devansh Arpit 3, Nicolas Ballas 4, Asja Fischer 5, Yoshua Bengio 36, Amos Storkey 7 Abstract arxiv:1711.04623v2 [cs.lg]
More informationMachine Learning CS 4900/5900. Lecture 03. Razvan C. Bunescu School of Electrical Engineering and Computer Science
Machine Learning CS 4900/5900 Razvan C. Bunescu School of Electrical Engineering and Computer Science bunescu@ohio.edu Machine Learning is Optimization Parametric ML involves minimizing an objective function
More informationLarge-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima J. Nocedal with N. Keskar Northwestern University D. Mudigere INTEL P. Tang INTEL M. Smelyanskiy INTEL 1 Initial Remarks SGD
More informationwhat can deep learning learn from linear regression? Benjamin Recht University of California, Berkeley
what can deep learning learn from linear regression? Benjamin Recht University of California, Berkeley Collaborators Joint work with Samy Bengio, Moritz Hardt, Michael Jordan, Jason Lee, Max Simchowitz,
More informationOPTIMIZATION METHODS IN DEEP LEARNING
Tutorial outline OPTIMIZATION METHODS IN DEEP LEARNING Based on Deep Learning, chapter 8 by Ian Goodfellow, Yoshua Bengio and Aaron Courville Presented By Nadav Bhonker Optimization vs Learning Surrogate
More informationNormalization Techniques
Normalization Techniques Devansh Arpit Normalization Techniques 1 / 39 Table of Contents 1 Introduction 2 Motivation 3 Batch Normalization 4 Normalization Propagation 5 Weight Normalization 6 Layer Normalization
More informationDon t Decay the Learning Rate, Increase the Batch Size. Samuel L. Smith, Pieter-Jan Kindermans, Quoc V. Le December 9 th 2017
Don t Decay the Learning Rate, Increase the Batch Size Samuel L. Smith, Pieter-Jan Kindermans, Quoc V. Le December 9 th 2017 slsmith@ Google Brain Three related questions: What properties control generalization?
More informationSub-Sampled Newton Methods for Machine Learning. Jorge Nocedal
Sub-Sampled Newton Methods for Machine Learning Jorge Nocedal Northwestern University Goldman Lecture, Sept 2016 1 Collaborators Raghu Bollapragada Northwestern University Richard Byrd University of Colorado
More informationTheory of Deep Learning IIb: Optimization Properties of SGD
CBMM Memo No. 72 December 27, 217 Theory of Deep Learning IIb: Optimization Properties of SGD by Chiyuan Zhang 1 Qianli Liao 1 Alexander Rakhlin 2 Brando Miranda 1 Noah Golowich 1 Tomaso Poggio 1 1 Center
More informationDeep Learning & Artificial Intelligence WS 2018/2019
Deep Learning & Artificial Intelligence WS 2018/2019 Linear Regression Model Model Error Function: Squared Error Has no special meaning except it makes gradients look nicer Prediction Ground truth / target
More informationDay 3 Lecture 3. Optimizing deep networks
Day 3 Lecture 3 Optimizing deep networks Convex optimization A function is convex if for all α [0,1]: f(x) Tangent line Examples Quadratics 2-norms Properties Local minimum is global minimum x Gradient
More informationCSCI 1951-G Optimization Methods in Finance Part 12: Variants of Gradient Descent
CSCI 1951-G Optimization Methods in Finance Part 12: Variants of Gradient Descent April 27, 2018 1 / 32 Outline 1) Moment and Nesterov s accelerated gradient descent 2) AdaGrad and RMSProp 4) Adam 5) Stochastic
More informationStochastic Optimization Methods for Machine Learning. Jorge Nocedal
Stochastic Optimization Methods for Machine Learning Jorge Nocedal Northwestern University SIAM CSE, March 2017 1 Collaborators Richard Byrd R. Bollagragada N. Keskar University of Colorado Northwestern
More informationRegularization and Optimization of Backpropagation
Regularization and Optimization of Backpropagation The Norwegian University of Science and Technology (NTNU) Trondheim, Norway keithd@idi.ntnu.no October 17, 2017 Regularization Definition of Regularization
More informationCSC321 Lecture 8: Optimization
CSC321 Lecture 8: Optimization Roger Grosse Roger Grosse CSC321 Lecture 8: Optimization 1 / 26 Overview We ve talked a lot about how to compute gradients. What do we actually do with them? Today s lecture:
More informationBased on the original slides of Hung-yi Lee
Based on the original slides of Hung-yi Lee Google Trends Deep learning obtains many exciting results. Can contribute to new Smart Services in the Context of the Internet of Things (IoT). IoT Services
More informationCSC321 Lecture 7: Optimization
CSC321 Lecture 7: Optimization Roger Grosse Roger Grosse CSC321 Lecture 7: Optimization 1 / 25 Overview We ve talked a lot about how to compute gradients. What do we actually do with them? Today s lecture:
More informationUnraveling the mysteries of stochastic gradient descent on deep neural networks
Unraveling the mysteries of stochastic gradient descent on deep neural networks Pratik Chaudhari UCLA VISION LAB 1 The question measures disagreement of predictions with ground truth Cat Dog... x = argmin
More informationNonlinear Optimization Methods for Machine Learning
Nonlinear Optimization Methods for Machine Learning Jorge Nocedal Northwestern University University of California, Davis, Sept 2018 1 Introduction We don t really know, do we? a) Deep neural networks
More informationImportance Reweighting Using Adversarial-Collaborative Training
Importance Reweighting Using Adversarial-Collaborative Training Yifan Wu yw4@andrew.cmu.edu Tianshu Ren tren@andrew.cmu.edu Lidan Mu lmu@andrew.cmu.edu Abstract We consider the problem of reweighting a
More informationAdvanced computational methods X Selected Topics: SGD
Advanced computational methods X071521-Selected Topics: SGD. In this lecture, we look at the stochastic gradient descent (SGD) method 1 An illustrating example The MNIST is a simple dataset of variety
More informationLecture 3: Minimizing Large Sums. Peter Richtárik
Lecture 3: Minimizing Large Sums Peter Richtárik Graduate School in Systems, Op@miza@on, Control and Networks Belgium 2015 Mo@va@on: Machine Learning & Empirical Risk Minimiza@on Training Linear Predictors
More informationNew Insights and Perspectives on the Natural Gradient Method
1 / 18 New Insights and Perspectives on the Natural Gradient Method Yoonho Lee Department of Computer Science and Engineering Pohang University of Science and Technology March 13, 2018 Motivation 2 / 18
More informationNeural Network Training
Neural Network Training Sargur Srihari Topics in Network Training 0. Neural network parameters Probabilistic problem formulation Specifying the activation and error functions for Regression Binary classification
More informationSVRG++ with Non-uniform Sampling
SVRG++ with Non-uniform Sampling Tamás Kern András György Department of Electrical and Electronic Engineering Imperial College London, London, UK, SW7 2BT {tamas.kern15,a.gyorgy}@imperial.ac.uk Abstract
More informationStochastic Variance Reduction for Nonconvex Optimization. Barnabás Póczos
1 Stochastic Variance Reduction for Nonconvex Optimization Barnabás Póczos Contents 2 Stochastic Variance Reduction for Nonconvex Optimization Joint work with Sashank Reddi, Ahmed Hefny, Suvrit Sra, and
More informationAdam: A Method for Stochastic Optimization
Adam: A Method for Stochastic Optimization Diederik P. Kingma, Jimmy Ba Presented by Content Background Supervised ML theory and the importance of optimum finding Gradient descent and its variants Limitations
More informationOptimization for neural networks
0 - : Optimization for neural networks Prof. J.C. Kao, UCLA Optimization for neural networks We previously introduced the principle of gradient descent. Now we will discuss specific modifications we make
More informationOverview of gradient descent optimization algorithms. HYUNG IL KOO Based on
Overview of gradient descent optimization algorithms HYUNG IL KOO Based on http://sebastianruder.com/optimizing-gradient-descent/ Problem Statement Machine Learning Optimization Problem Training samples:
More informationBlack-box α-divergence Minimization
Black-box α-divergence Minimization José Miguel Hernández-Lobato, Yingzhen Li, Daniel Hernández-Lobato, Thang Bui, Richard Turner, Harvard University, University of Cambridge, Universidad Autónoma de Madrid.
More informationECE G: Special Topics in Signal Processing: Sparsity, Structure, and Inference
ECE 18-898G: Special Topics in Signal Processing: Sparsity, Structure, and Inference Neural Networks: A brief touch Yuejie Chi Department of Electrical and Computer Engineering Spring 2018 1/41 Outline
More informationNon-Convex Optimization in Machine Learning. Jan Mrkos AIC
Non-Convex Optimization in Machine Learning Jan Mrkos AIC The Plan 1. Introduction 2. Non convexity 3. (Some) optimization approaches 4. Speed and stuff? Neural net universal approximation Theorem (1989):
More informationMini-Course 1: SGD Escapes Saddle Points
Mini-Course 1: SGD Escapes Saddle Points Yang Yuan Computer Science Department Cornell University Gradient Descent (GD) Task: min x f (x) GD does iterative updates x t+1 = x t η t f (x t ) Gradient Descent
More informationIntroduction to Neural Networks
CUONG TUAN NGUYEN SEIJI HOTTA MASAKI NAKAGAWA Tokyo University of Agriculture and Technology Copyright by Nguyen, Hotta and Nakagawa 1 Pattern classification Which category of an input? Example: Character
More informationNegative Momentum for Improved Game Dynamics
Negative Momentum for Improved Game Dynamics Gauthier Gidel Reyhane Askari Hemmat Mohammad Pezeshki Gabriel Huang Rémi Lepriol Simon Lacoste-Julien Ioannis Mitliagkas Mila & DIRO, Université de Montréal
More informationDeep Relaxation: PDEs for optimizing Deep Neural Networks
Deep Relaxation: PDEs for optimizing Deep Neural Networks IPAM Mean Field Games August 30, 2017 Adam Oberman (McGill) Thanks to the Simons Foundation (grant 395980) and the hospitality of the UCLA math
More informationOptimization for Training I. First-Order Methods Training algorithm
Optimization for Training I First-Order Methods Training algorithm 2 OPTIMIZATION METHODS Topics: Types of optimization methods. Practical optimization methods breakdown into two categories: 1. First-order
More informationarxiv: v2 [cs.cv] 8 Mar 2018
RESIDUAL CONNECTIONS ENCOURAGE ITERATIVE IN- FERENCE Stanisław Jastrzębski 1,2,, Devansh Arpit 2,, Nicolas Ballas 3, Vikas Verma 5, Tong Che 2 & Yoshua Bengio 2,6 arxiv:171773v2 [cs.cv] 8 Mar 2018 1 Jagiellonian
More informationECS171: Machine Learning
ECS171: Machine Learning Lecture 4: Optimization (LFD 3.3, SGD) Cho-Jui Hsieh UC Davis Jan 22, 2018 Gradient descent Optimization Goal: find the minimizer of a function min f (w) w For now we assume f
More informationBased on the original slides of Hung-yi Lee
Based on the original slides of Hung-yi Lee New Activation Function Rectified Linear Unit (ReLU) σ z a a = z Reason: 1. Fast to compute 2. Biological reason a = 0 [Xavier Glorot, AISTATS 11] [Andrew L.
More informationIMPROVING STOCHASTIC GRADIENT DESCENT
IMPROVING STOCHASTIC GRADIENT DESCENT WITH FEEDBACK Jayanth Koushik & Hiroaki Hayashi Language Technologies Institute Carnegie Mellon University Pittsburgh, PA 15213, USA {jkoushik,hiroakih}@cs.cmu.edu
More informationMachine Learning and Data Mining. Linear regression. Prof. Alexander Ihler
+ Machine Learning and Data Mining Linear regression Prof. Alexander Ihler Supervised learning Notation Features x Targets y Predictions ŷ Parameters θ Learning algorithm Program ( Learner ) Change µ Improve
More informationOn the saddle point problem for non-convex optimization
On the saddle point problem for non-convex optimization Razvan Pascanu Université de Montréal r.pascanu@gmail.com Surya Ganguli Stanford University sganguli@standford.edu Yann N. Dauphin Université de
More informationCredit Assignment: Beyond Backpropagation
Credit Assignment: Beyond Backpropagation Yoshua Bengio 11 December 2016 AutoDiff NIPS 2016 Workshop oo b s res P IT g, M e n i arn nlin Le ain o p ee em : D will r G PLU ters p cha k t, u o is Deep Learning
More informationGRADIENT DESCENT. CSE 559A: Computer Vision GRADIENT DESCENT GRADIENT DESCENT [0, 1] Pr(y = 1) w T x. 1 f (x; θ) = 1 f (x; θ) = exp( w T x)
0 x x x CSE 559A: Computer Vision For Binary Classification: [0, ] f (x; ) = σ( x) = exp( x) + exp( x) Output is interpreted as probability Pr(y = ) x are the log-odds. Fall 207: -R: :30-pm @ Lopata 0
More informationNeed for Deep Networks Perceptron. Can only model linear functions. Kernel Machines. Non-linearity provided by kernels
Need for Deep Networks Perceptron Can only model linear functions Kernel Machines Non-linearity provided by kernels Need to design appropriate kernels (possibly selecting from a set, i.e. kernel learning)
More informationTHE ANISOTROPIC NOISE IN STOCHASTIC GRADIENT DESCENT: ITS BEHAVIOR OF ESCAPING FROM MIN-
THE ANISOTROPIC NOISE IN STOCHASTIC GRADIENT DESCENT: ITS BEHAVIOR OF ESCAPING FROM MIN- IMA AND REGULARIZATION EFFECTS Anonymous authors Paper under double-blind review ABSTRACT Understanding the behavior
More informationAn Evolving Gradient Resampling Method for Machine Learning. Jorge Nocedal
An Evolving Gradient Resampling Method for Machine Learning Jorge Nocedal Northwestern University NIPS, Montreal 2015 1 Collaborators Figen Oztoprak Stefan Solntsev Richard Byrd 2 Outline 1. How to improve
More informationTips for Deep Learning
Tips for Deep Learning Recipe of Deep Learning Step : define a set of function Step : goodness of function Step 3: pick the best function NO Overfitting! NO YES Good Results on Testing Data? YES Good Results
More informationNeural Networks: Optimization & Regularization
Neural Networks: Optimization & Regularization Shan-Hung Wu shwu@cs.nthu.edu.tw Department of Computer Science, National Tsing Hua University, Taiwan Machine Learning Shan-Hung Wu (CS, NTHU) NN Opt & Reg
More informationStochastic Optimization Algorithms Beyond SG
Stochastic Optimization Algorithms Beyond SG Frank E. Curtis 1, Lehigh University involving joint work with Léon Bottou, Facebook AI Research Jorge Nocedal, Northwestern University Optimization Methods
More informationComparison of Modern Stochastic Optimization Algorithms
Comparison of Modern Stochastic Optimization Algorithms George Papamakarios December 214 Abstract Gradient-based optimization methods are popular in machine learning applications. In large-scale problems,
More informationNeed for Deep Networks Perceptron. Can only model linear functions. Kernel Machines. Non-linearity provided by kernels
Need for Deep Networks Perceptron Can only model linear functions Kernel Machines Non-linearity provided by kernels Need to design appropriate kernels (possibly selecting from a set, i.e. kernel learning)
More informationTsybakov noise adap/ve margin- based ac/ve learning
Tsybakov noise adap/ve margin- based ac/ve learning Aar$ Singh A. Nico Habermann Associate Professor NIPS workshop on Learning Faster from Easy Data II Dec 11, 2015 Passive Learning Ac/ve Learning (X j,?)
More informationMachine Learning. Lecture 04: Logistic and Softmax Regression. Nevin L. Zhang
Machine Learning Lecture 04: Logistic and Softmax Regression Nevin L. Zhang lzhang@cse.ust.hk Department of Computer Science and Engineering The Hong Kong University of Science and Technology This set
More informationLeast Mean Squares Regression. Machine Learning Fall 2017
Least Mean Squares Regression Machine Learning Fall 2017 1 Lecture Overview Linear classifiers What func?ons do linear classifiers express? Least Squares Method for Regression 2 Where are we? Linear classifiers
More informationECE521 lecture 4: 19 January Optimization, MLE, regularization
ECE521 lecture 4: 19 January 2017 Optimization, MLE, regularization First four lectures Lectures 1 and 2: Intro to ML Probability review Types of loss functions and algorithms Lecture 3: KNN Convexity
More informationThe Deep Ritz method: A deep learning-based numerical algorithm for solving variational problems
The Deep Ritz method: A deep learning-based numerical algorithm for solving variational problems Weinan E 1 and Bing Yu 2 arxiv:1710.00211v1 [cs.lg] 30 Sep 2017 1 The Beijing Institute of Big Data Research,
More informationNon-Convex Optimization. CS6787 Lecture 7 Fall 2017
Non-Convex Optimization CS6787 Lecture 7 Fall 2017 First some words about grading I sent out a bunch of grades on the course management system Everyone should have all their grades in Not including paper
More informationLecture 9: Generalization
Lecture 9: Generalization Roger Grosse 1 Introduction When we train a machine learning model, we don t just want it to learn to model the training data. We want it to generalize to data it hasn t seen
More informationA picture of the energy landscape of! deep neural networks
A picture of the energy landscape of! deep neural networks Pratik Chaudhari December 15, 2017 UCLA VISION LAB 1 Dy (x; w) = (w p (w p 1 (... (w 1 x))...)) w = argmin w (x,y ) 2 D kx 1 {y =i } log Dy i
More informationDeep Feedforward Networks
Deep Feedforward Networks Liu Yang March 30, 2017 Liu Yang Short title March 30, 2017 1 / 24 Overview 1 Background A general introduction Example 2 Gradient based learning Cost functions Output Units 3
More informationTopics we covered. Machine Learning. Statistics. Optimization. Systems! Basics of probability Tail bounds Density Estimation Exponential Families
Midterm Review Topics we covered Machine Learning Optimization Basics of optimization Convexity Unconstrained: GD, SGD Constrained: Lagrange, KKT Duality Linear Methods Perceptrons Support Vector Machines
More informationDeep Learning & Neural Networks Lecture 4
Deep Learning & Neural Networks Lecture 4 Kevin Duh Graduate School of Information Science Nara Institute of Science and Technology Jan 23, 2014 2/20 3/20 Advanced Topics in Optimization Today we ll briefly
More informationLinear Discrimination Functions
Laurea Magistrale in Informatica Nicola Fanizzi Dipartimento di Informatica Università degli Studi di Bari November 4, 2009 Outline Linear models Gradient descent Perceptron Minimum square error approach
More informationBayesian Deep Learning
Bayesian Deep Learning Mohammad Emtiyaz Khan RIKEN Center for AI Project, Tokyo http://emtiyaz.github.io Keywords Bayesian Statistics Gaussian distribution, Bayes rule Continues Optimization Gradient descent,
More informationOn the fast convergence of random perturbations of the gradient flow.
On the fast convergence of random perturbations of the gradient flow. Wenqing Hu. 1 (Joint work with Chris Junchi Li 2.) 1. Department of Mathematics and Statistics, Missouri S&T. 2. Department of Operations
More informationCS 6140: Machine Learning Spring What We Learned Last Week 2/26/16
Logis@cs CS 6140: Machine Learning Spring 2016 Instructor: Lu Wang College of Computer and Informa@on Science Northeastern University Webpage: www.ccs.neu.edu/home/luwang Email: luwang@ccs.neu.edu Sign
More informationCOMPARING FIXED AND ADAPTIVE COMPUTATION TIME FOR RE-
Workshop track - ICLR COMPARING FIXED AND ADAPTIVE COMPUTATION TIME FOR RE- CURRENT NEURAL NETWORKS Daniel Fojo, Víctor Campos, Xavier Giró-i-Nieto Universitat Politècnica de Catalunya, Barcelona Supercomputing
More informationCS 6140: Machine Learning Spring What We Learned Last Week. Survey 2/26/16. VS. Model
Logis@cs CS 6140: Machine Learning Spring 2016 Instructor: Lu Wang College of Computer and Informa@on Science Northeastern University Webpage: www.ccs.neu.edu/home/luwang Email: luwang@ccs.neu.edu Assignment
More informationFast yet Simple Natural-Gradient Variational Inference in Complex Models
Fast yet Simple Natural-Gradient Variational Inference in Complex Models Mohammad Emtiyaz Khan RIKEN Center for AI Project, Tokyo, Japan http://emtiyaz.github.io Joint work with Wu Lin (UBC) DidrikNielsen,
More informationStochastic Optimization with Variance Reduction for Infinite Datasets with Finite Sum Structure
Stochastic Optimization with Variance Reduction for Infinite Datasets with Finite Sum Structure Alberto Bietti Julien Mairal Inria Grenoble (Thoth) March 21, 2017 Alberto Bietti Stochastic MISO March 21,
More informationCS 6140: Machine Learning Spring 2016
CS 6140: Machine Learning Spring 2016 Instructor: Lu Wang College of Computer and Informa?on Science Northeastern University Webpage: www.ccs.neu.edu/home/luwang Email: luwang@ccs.neu.edu Logis?cs Assignment
More informationData Mining & Machine Learning
Data Mining & Machine Learning CS57300 Purdue University March 1, 2018 1 Recap of Last Class (Model Search) Forward and Backward passes 2 Feedforward Neural Networks Neural Networks: Architectures 2-layer
More informationLearning Long Term Dependencies with Gradient Descent is Difficult
Learning Long Term Dependencies with Gradient Descent is Difficult IEEE Trans. on Neural Networks 1994 Yoshua Bengio, Patrice Simard, Paolo Frasconi Presented by: Matt Grimes, Ayse Naz Erkan Recurrent
More informationTips for Deep Learning
Tips for Deep Learning Recipe of Deep Learning Step : define a set of function Step : goodness of function Step 3: pick the best function NO Overfitting! NO YES Good Results on Testing Data? YES Good Results
More informationApprentissage, réseaux de neurones et modèles graphiques (RCP209) Neural Networks and Deep Learning
Apprentissage, réseaux de neurones et modèles graphiques (RCP209) Neural Networks and Deep Learning Nicolas Thome Prenom.Nom@cnam.fr http://cedric.cnam.fr/vertigo/cours/ml2/ Département Informatique Conservatoire
More informationComments. Assignment 3 code released. Thought questions 3 due this week. Mini-project: hopefully you have started. implement classification algorithms
Neural networks Comments Assignment 3 code released implement classification algorithms use kernels for census dataset Thought questions 3 due this week Mini-project: hopefully you have started 2 Example:
More informationDeep Neural Networks: From Flat Minima to Numerically Nonvacuous Generalization Bounds via PAC-Bayes
Deep Neural Networks: From Flat Minima to Numerically Nonvacuous Generalization Bounds via PAC-Bayes Daniel M. Roy University of Toronto; Vector Institute Joint work with Gintarė K. Džiugaitė University
More informationUNDERSTANDING LOCAL MINIMA IN NEURAL NET-
UNDERSTANDING LOCAL MINIMA IN NEURAL NET- WORKS BY LOSS SURFACE DECOMPOSITION Anonymous authors Paper under double-blind review ABSTRACT To provide principled ways of designing proper Deep Neural Network
More informationLecture 6 Optimization for Deep Neural Networks
Lecture 6 Optimization for Deep Neural Networks CMSC 35246: Deep Learning Shubhendu Trivedi & Risi Kondor University of Chicago April 12, 2017 Things we will look at today Stochastic Gradient Descent Things
More informationNumerical Optimization Techniques
Numerical Optimization Techniques Léon Bottou NEC Labs America COS 424 3/2/2010 Today s Agenda Goals Representation Capacity Control Operational Considerations Computational Considerations Classification,
More informationAsynchronous Mini-Batch Gradient Descent with Variance Reduction for Non-Convex Optimization
Proceedings of the hirty-first AAAI Conference on Artificial Intelligence (AAAI-7) Asynchronous Mini-Batch Gradient Descent with Variance Reduction for Non-Convex Optimization Zhouyuan Huo Dept. of Computer
More informationarxiv: v2 [math.oc] 5 May 2018
The Impact of Local Geometry and Batch Size on Stochastic Gradient Descent for Nonconvex Problems Viva Patel a a Department of Statistics, University of Chicago, Illinois, USA arxiv:1709.04718v2 [math.oc]
More informationNormalization Techniques in Training of Deep Neural Networks
Normalization Techniques in Training of Deep Neural Networks Lei Huang ( 黄雷 ) State Key Laboratory of Software Development Environment, Beihang University Mail:huanglei@nlsde.buaa.edu.cn August 17 th,
More informationLogistic Regression. Stochastic Gradient Descent
Tutorial 8 CPSC 340 Logistic Regression Stochastic Gradient Descent Logistic Regression Model A discriminative probabilistic model for classification e.g. spam filtering Let x R d be input and y { 1, 1}
More informationStochastic Gradient Descent. CS 584: Big Data Analytics
Stochastic Gradient Descent CS 584: Big Data Analytics Gradient Descent Recap Simplest and extremely popular Main Idea: take a step proportional to the negative of the gradient Easy to implement Each iteration
More informationMore Optimization. Optimization Methods. Methods
More More Optimization Optimization Methods Methods Yann YannLeCun LeCun Courant CourantInstitute Institute http://yann.lecun.com http://yann.lecun.com (almost) (almost) everything everything you've you've
More informationLarge-scale Stochastic Optimization
Large-scale Stochastic Optimization 11-741/641/441 (Spring 2016) Hanxiao Liu hanxiaol@cs.cmu.edu March 24, 2016 1 / 22 Outline 1. Gradient Descent (GD) 2. Stochastic Gradient Descent (SGD) Formulation
More informationSimple Techniques for Improving SGD. CS6787 Lecture 2 Fall 2017
Simple Techniques for Improving SGD CS6787 Lecture 2 Fall 2017 Step Sizes and Convergence Where we left off Stochastic gradient descent x t+1 = x t rf(x t ; yĩt ) Much faster per iteration than gradient
More informationarxiv: v2 [math.oc] 1 Nov 2017
Stochastic Non-convex Optimization with Strong High Probability Second-order Convergence arxiv:1710.09447v [math.oc] 1 Nov 017 Mingrui Liu, Tianbao Yang Department of Computer Science The University of
More informationForward and Reverse Gradient-based Hyperparameter Optimization
Forward and Reverse Gradient-based Hyperparameter Optimization Luca Franceschi 1,2, Michele Donini 1, Paolo Frasconi 3 and Massimilano Pontil 1,2 massimiliano.pontil@iit.it (1) Istituto Italiano di Tecnologia,
More informationProvable Non-Convex Min-Max Optimization
Provable Non-Convex Min-Max Optimization Mingrui Liu, Hassan Rafique, Qihang Lin, Tianbao Yang Department of Computer Science, The University of Iowa, Iowa City, IA, 52242 Department of Mathematics, The
More informationContents. 1 Introduction. 1.1 History of Optimization ALG-ML SEMINAR LISSA: LINEAR TIME SECOND-ORDER STOCHASTIC ALGORITHM FEBRUARY 23, 2016
ALG-ML SEMINAR LISSA: LINEAR TIME SECOND-ORDER STOCHASTIC ALGORITHM FEBRUARY 23, 2016 LECTURERS: NAMAN AGARWAL AND BRIAN BULLINS SCRIBE: KIRAN VODRAHALLI Contents 1 Introduction 1 1.1 History of Optimization.....................................
More informationTTIC 31230, Fundamentals of Deep Learning David McAllester, Winter Multiclass Logistic Regression. Multilayer Perceptrons (MLPs)
TTIC 31230, Fundamentals of Deep Learning David McAllester, Winter 2018 Multiclass Logistic Regression Multilayer Perceptrons (MLPs) Stochastic Gradient Descent (SGD) 1 Multiclass Classification We consider
More informationLearning Deep Architectures
Learning Deep Architectures Yoshua Bengio, U. Montreal Microsoft Cambridge, U.K. July 7th, 2009, Montreal Thanks to: Aaron Courville, Pascal Vincent, Dumitru Erhan, Olivier Delalleau, Olivier Breuleux,
More informationRecurrent Neural Network Training with Preconditioned Stochastic Gradient Descent
Recurrent Neural Network Training with Preconditioned Stochastic Gradient Descent 1 Xi-Lin Li, lixilinx@gmail.com arxiv:1606.04449v2 [stat.ml] 8 Dec 2016 Abstract This paper studies the performance of
More informationIntroduction to Machine Learning HW6
CS 189 Spring 2018 Introduction to Machine Learning HW6 Your self-grade URL is http://eecs189.org/self_grade?question_ids=1_1,1_ 2,2_1,2_2,3_1,3_2,3_3,4_1,4_2,4_3,4_4,4_5,4_6,5_1,5_2,6. This homework is
More information