Deep unsupervised learning
|
|
- Virgil Morrison
- 5 years ago
- Views:
Transcription
1 Deep unsupervised learning Advanced data-mining Yongdai Kim Department of Statistics, Seoul National University, South Korea
2 Unsupervised learning In machine learning, there are 3 kinds of learning paradigm. 1. Supervised learning 2. Unsupervised learning 3. Reinforcement learning In this chapter, we only consider unsupervised learning. What is unsupervised learning? The problem of unsupervised learning is that of trying to find hidden structure in unlabeled data. Example clustering(k-means clustering, mixture of gaussian, etc.) factor analysis(pca, etc.)
3 Unsupervised learning in deep learning Stochastic case Restricted Boltzmann Machines Sigmoid belief networks Deep Boltzmann Machines Deep Belief Networks Deterministic case Auto Encoders Denoising Auto Encoders Contractive Auto Encoders
4 Restricted Boltzmann Machines A generative stochastic model that can learn a probability distribution over its set of inputs. Initially invented under the name Harmonium(P.Smolensky, 1986). Rose to prominence after Hinton and his collaborators started deep learning with two important papers in 2006.(G.E.Hinton et al., 2006, G.E.Hinton and R.R.Salakhutdinov, 2006) They can be trained in either supervised or unsupervised ways, depending on the task. Actually, RBM is not a deep structure, but it is a building block of deep structures.
5 Restricted Boltzmann Machines As their name implied, RBM are a variant of Boltzmann machines, with the restriction that their neurons must form a bipartite graph. Figure: Structures of BM and RBM
6 Restricted Boltzmann Machines Structure of RBM RBM has binary-valued hidden and visible units. v = (v 1,..., v m ) : visible units. h = (h 1,..., h n ) : hidden units. Given these, the energy of (v, h) is defined as: E(v, h) = b v c h v Wh = b i v i c j h j i j i w ij v i h i. j where b and c are called bias vector and W is called weight matrix. Parameter θ := (W, b, c).
7 Restricted Boltzmann Machines The probability distributions over hidden and visible vectors are defined in terms of the energy function: P (v, h) = 1 Z e E(v,h) where Z is called partition function, just a normalizing constant.(i.e. Z = v h e E(v,h) ) Figure: Example of RBM
8 Learning Restricted Boltzmann Machines It is easy to calculate conditional distribution, therefore, calculating posterior distribution is easy. The conditional probability of visible units v given h is : P (v h) = i P (v i h) where P (v i h) = sigm(b i + j w ij h j ). Conversely, the conditional probability of h given v is : P (h v) = j P (h j v) where P (h j v) = sigm(c j + i w ij v i ).
9 Learning Restricted Boltzmann Machines The basic idea is to minimize the minus log-likelihood function. Calculating minimizer of log P (v) = log h P (v, h) is hard, so we use gradient descent algorithm. [ log P (v)] θ = h = h E(v, h) P (h v) E(ṽ, h) θ θ ṽ,h E(v, h) E(ṽ, h) P (h v) Eṽ,h [ ] θ θ It is very slow to compute the above partial derivations, because an exponential number of sum operations is needed in computing Eṽ,h [ E(ṽ,h) θ ].
10 Learning Restricted Boltzmann Machines Contrastive Divergence algorithm To approximate these partial derivatives efficiently, the contrastive divergence (CD) algorithm, especially k-step contrastive divergence (CD-k), could be employed as a standard method for training RBMs. It can be seen that the core of the CD-k learning algorithm is a special case of Gibbs sampling, which is a Markov chain Monte Carlo (MCMC) algorithm for obtaining a sequence of observations which are approximated from a specified multivariate probability distribution when direct sampling is difficult.
11 Learning Restricted Boltzmann Machines For k Gibbs steps starting from a training example(i.e., sampling from ˆP ): v 0 ˆP h 0 P (h v 0 ) v 1 P (v h 0 ) h 1 P (h v 1 ). v k P (v h k 1 ) It is easy to sample from conditional distribution because of the structure of RBM.
12 Learning Restricted Boltzmann Machines Using Gibbs chain, we can appropriately compute the gradient of log-likelihood with respect to θ at the training sample v, log P (v) θ h P (h v) E(v, h) θ h P (h v k ) E(v k, h). θ Figure: Contrastive Divergence
13 Deep generative models Deep generative model is a generative model that has more than 2 hidden layers. Deep models often have millions of parameters. There are three different kinds of deep generative models: directed, undirected and mixed. We study directed and undirected version briefly, and discuss about mixed model in depth.
14 Deep directed networks The bottom level contains the visible data, and the remaining layers are hidden. If all the nodes are binary, this is called a sigmoid belief net.(r.neal, 1992) Figure: Sigmoid belief network
15 Deep directed networks In this case, the model defines the following joint distribution: P (v, h (1),..., h l ) = P (v h (1) )P (h (2) h (1) ) P (h (l 1) h (l) )P (h (l) ) P (h (k) h (k+1) ) = n k i=1 P (h (k) i h (k+1) ) P (h (k) i = 1 h (k+1) ) = sigm(b (k+1) i + j W (k+1) ij h (k+1) k ) and where h (0) = v. n l P (h (l) ) = Ber(h (l) i w (l) i ). i=1
16 Deep directed networks Unfortunately, inference in directed models is intractable because the posterior on the hidden nodes is correlated due to explaining away. One can use mean field approximations, but these may not be very accurate, since they approximate the correlated posterior with a factorial posterior. Remark Explaining away means that a prior independent causes of an event can become nonindependent given the observation of the event.
17 Deep Boltzmann Machines A natural alternative to a directed model is to construct a deep undirected model. For example, we can stack a series of RBMs on top of each other, and this is known as a deep boltzmann machine(dbm) (R.R.Salakhutdinov and G.E.Hinton, 2009). The energy function of a DBM which has l hidden layers can be defined as: l l E(v, h (1),..., h (l) ) = h (k 1)T W (k) h (k) b (k)t h (k) k=1 k=0 where h (0) = v. Correspondingly, the joint distribution can be defined as: P (v, h (1),..., h (l) ) = 1 Z e E(v,h(1),...,h (l) ) where Z is a partition function.
18 Deep Boltzmann Machines The main advantage of DBM is that one can perform efficient layer-wise Gibbs sampling since all the nodes in each layer are conditionally independent of each other given the layers above and below. But training undirected models is more difficult, because the partition function Z is intractable to calculate. Figure: 3-hidden layers DBM
19 Deep Belief Networks Deep belief networks(g.e.hinton et al., 2006) is a hybrid graph model consisting of both undirected and directed parts. The top two layers make up an undirected graph of an associative memory network, which can also be viewed as an RBM. The other layers make up a directed graph. Figure: 3-hidden layers DBN
20 Deep Belief Networks A DBN with l layers models the joint distribution between observed variables v and hidden layers h (k), k = 1,..., l, as follows: p(v, h (1),..., h (l) ) = P (v h (1) ) P (h (l 2) h (l 1) )P (h (l 1), h (l) ) P (h (k) h (k+1) ) = i P (h (k) i h (k+1) ) P (h (k) i h (k+1) ) = sigm(b (k+1) i + j W (k+1) ij h (k+1) j ) and P (h (l 1), h (l) ) is a RBM.
21 Deep Belief Networks Figure: The structure of DBN
22 Learning Deep Belief Networks The learning procedure of a DBN can also be divided into two stages : pre-training and fine-tuning.(g.e.hinton et al., 2006) Unsupervised pre-training provides a good initialization of the network. Supervised fine-tuning Up-down algorithm We only discuss unsupervised pre-training algorithm.
23 Learning Deep Belief Networks First step: Construct an RBM with an input layer v and a hidden layer h. Train the RBM Second step: Stack another hidden layer on top of the RBM to form a new RBM Fix W (1), sample h (1) from Q(h (1) v) as input. Train W (2) as RBM. Third step: Continue to stack layers on top of the network, train it as previous step, with sample sampled from Q(h (2) h (1) ). And so on...
24 Learning Deep Belief Networks Why greedy training works? It was shown that greedy training process maximizes the lower-bound of the log-likelihood of the data. (G.E.Hinton et al., 2006) Figure: DBN Greedy training
25 Learning Deep Belief Networks Let s consider 2-hidden layer DBN. A DBN with two hidden layers and tied weight is equivalent to an RBM. (i.e. P (v, h 1 W 1 ) = h 2 P (v, h1, h 2 W 1 ) is RBM)(G.E.Hinton et al., 2006) Figure: tied DBN equals RBM
26 Learning Deep Belief Networks 1 So, if we constrain W 2 = W 1 T, then P (v, h 1 W 1 ) would be a RBM. Find optimal Ŵ 1 such that log P (v Ŵ1 ) = max W 1 log P (v W 1 ) using CD algorithm. Note that max W 1,W 2 P (v W1, W 2 ) max W 1 P (v W 1 ). 2 P (h 1, h 2 Ŵ 1 ) is a RBM, so untie the weight Ŵ 1 to increase P (h 1 ). Find optimal Ŵ 2 such that log P (h 1 Ŵ2 ) = max W 2 log P (h 1 W 2 ) using CD algorithm. It makes increase of lower bound of original log-likelihood.
27 Learning Deep Belief Networks However, in practice, we want to be able to use any number of hidden units in each level. This means we will not be able to initialize the weights so that W l = W l 1T and this voids the theoretical guarantee. Nevertheless the method works well in practice.
28 Auto-Encoders An AE can be used for dimensionality reduction of high-dimensional data. The standard AE is a symmetric multilayer feed-forward network and desired output of an AE should be identical to the input for learning the identity mapping and extracting unsupervised features. Figure: An example of AE
29 Auto-Encoders By training, An AE can not only generate a hidden representation from an input, but also reconstruct the output as the input from the hidden representation. The part from the input to the interlayer is called encoder, and the part from the interlayer to the output is called decoder. Figure: AE with encoder and decoder
30 Auto-Encoders Encoding refers to the process of mapping an input v R m to a hidden representation h(v) R n m namely, h(v) = σ(wv + b) where W R m n is a weight matrix, b R n is a bias vector, and σ(v) is sigmoid function. Decoding refers to the process of mapping a hidden representation h(v) to the output o for reconstructing the input v, o = σ( Wh(v) + b) where W R n m is a weight matrix, and b R m is a bias vector.
31 Auto-Encoders W is not necessarily equal to W T. If W = W T, then we call the weights are tied. Figure: A single hidden layer AE
32 Stacked AE(SAE) We can construct a deep AE that has an input layer, 2l 1 hidden layers and an output layer: h (1) = σ(w (1) v) h (k) = σ(w (k) h (k 1) + b (k) ), 2 k 2l 1 o = σ(w (2l) h (2l 1) + b (2l) ) where W (k) are weight matrices and b (k) are bias vectors. Figure: A multi hidden layer AE
33 Learning AE and SAE As a special kind of multilayer neural network, the weights and biases of an AE can be trained using back propagation algorithm. However, the BP algorithm may produce very different results from distinct initializations of weights and biases in deep cases, generally converging with relatively slow speed, or even not converging at all.
34 Learning AE and SAE An effective strategy for solving the problem of local minima is to adopt the following two-stage training approach.(g.e.hinton and R.R.Salakhutdinov, 2006) Unsupervised pre-training: Take every two adjacent layers as an RBM with its output as the input of the next higher RBM, and train all these layer-wise RBMs by an appropriate method, such as CD learning. Supervised fine-tuning: after unsupervised pre-training, fine-tune all the network parameters by using BP algorithm.
35 Learning AE and SAE Figure: Training process of SAE
36 Denoising Auto-Encoders A DAE(P.Vincent et al., 2008) is obtained by training AE in a special way, which can have some resistant ability to noise and more stable reconstruction of data. At first, it corrupts the raw input v to a partially destroyed version ṽ by means of a stochastic mapping ṽ q D (ṽ v). (e.g. set 25% of inputs to 0) Then, it takes ṽ as the input and v as the output to train the AE. Finally, it takes the output as a function of ṽ instead of v. Figure: A denoising AE
37 Contractive Auto-Encoders A CAE(S.Rifai et al., 2011) is similar to a DAE in function. It has a robust performance on small variations of training samples. Let assume there are N samples. The objective function of a CAE is: N N L(v i, o i ) + λ J f (v i ) 2 F i=1 where J f (v) 2 F = j,k i=1 ( hk (v j ) x j ) 2.
38 References G.E.Hinton, S.Osindero and Y.Teh. A fast learning algorithm for deep belief nets. Neural computation. 18(7). pp G.E.Hinton and R.R.Salakhutdinov. Reducing the dimensionality of data with neural networks. Science. 313(5786). pp H.Larochelle, D.Erhan, A.Courville, J.Bergstra and Y.Bengio. An empirical evaluation of deep architectures on problems with many factors of variation. Proceedings of the 24th international conference on Machine learning. pp R.Neal. Connectionist learning of belief networks. Artificial intelligence. 56(1). pp
39 References S.Rifai, P.Vincent, X.Muller, X.Glorot and Y.Bengio. Contractive auto-encoders: Explicit invariance during feature extraction. Proceedings of the 28th International Conference on Machine Learning (ICML-11). pp R.Salakhutdinov and G.E.Hinton. Deep boltzmann machines. International Conference on Artificial Intelligence and Statistics. pp P.Smolensky. Information processing in dynamical systems: Foundations of harmony theory. Department of Computer Science, University of Colorado, Boulder P.Vincent, H.Larochelle, Y.Bengio and P.Manzagol. Extracting and composing robust features with denoising autoencoders. Proceedings of the 25th international conference on Machine learning. pp
Greedy Layer-Wise Training of Deep Networks
Greedy Layer-Wise Training of Deep Networks Yoshua Bengio, Pascal Lamblin, Dan Popovici, Hugo Larochelle NIPS 2007 Presented by Ahmed Hefny Story so far Deep neural nets are more expressive: Can learn
More informationLecture 16 Deep Neural Generative Models
Lecture 16 Deep Neural Generative Models CMSC 35246: Deep Learning Shubhendu Trivedi & Risi Kondor University of Chicago May 22, 2017 Approach so far: We have considered simple models and then constructed
More informationUNSUPERVISED LEARNING
UNSUPERVISED LEARNING Topics Layer-wise (unsupervised) pre-training Restricted Boltzmann Machines Auto-encoders LAYER-WISE (UNSUPERVISED) PRE-TRAINING Breakthrough in 2006 Layer-wise (unsupervised) pre-training
More informationReading Group on Deep Learning Session 4 Unsupervised Neural Networks
Reading Group on Deep Learning Session 4 Unsupervised Neural Networks Jakob Verbeek & Daan Wynen 206-09-22 Jakob Verbeek & Daan Wynen Unsupervised Neural Networks Outline Autoencoders Restricted) Boltzmann
More informationA graph contains a set of nodes (vertices) connected by links (edges or arcs)
BOLTZMANN MACHINES Generative Models Graphical Models A graph contains a set of nodes (vertices) connected by links (edges or arcs) In a probabilistic graphical model, each node represents a random variable,
More informationThe Origin of Deep Learning. Lili Mou Jan, 2015
The Origin of Deep Learning Lili Mou Jan, 2015 Acknowledgment Most of the materials come from G. E. Hinton s online course. Outline Introduction Preliminary Boltzmann Machines and RBMs Deep Belief Nets
More informationUnsupervised Learning
CS 3750 Advanced Machine Learning hkc6@pitt.edu Unsupervised Learning Data: Just data, no labels Goal: Learn some underlying hidden structure of the data P(, ) P( ) Principle Component Analysis (Dimensionality
More informationKnowledge Extraction from DBNs for Images
Knowledge Extraction from DBNs for Images Son N. Tran and Artur d Avila Garcez Department of Computer Science City University London Contents 1 Introduction 2 Knowledge Extraction from DBNs 3 Experimental
More informationDeep Belief Networks are compact universal approximators
1 Deep Belief Networks are compact universal approximators Nicolas Le Roux 1, Yoshua Bengio 2 1 Microsoft Research Cambridge 2 University of Montreal Keywords: Deep Belief Networks, Universal Approximation
More informationDeep Learning Srihari. Deep Belief Nets. Sargur N. Srihari
Deep Belief Nets Sargur N. Srihari srihari@cedar.buffalo.edu Topics 1. Boltzmann machines 2. Restricted Boltzmann machines 3. Deep Belief Networks 4. Deep Boltzmann machines 5. Boltzmann machines for continuous
More informationLearning Deep Architectures for AI. Part II - Vijay Chakilam
Learning Deep Architectures for AI - Yoshua Bengio Part II - Vijay Chakilam Limitations of Perceptron x1 W, b 0,1 1,1 y x2 weight plane output =1 output =0 There is no value for W and b such that the model
More informationIntroduction to Restricted Boltzmann Machines
Introduction to Restricted Boltzmann Machines Ilija Bogunovic and Edo Collins EPFL {ilija.bogunovic,edo.collins}@epfl.ch October 13, 2014 Introduction Ingredients: 1. Probabilistic graphical models (undirected,
More informationChapter 20. Deep Generative Models
Peng et al.: Deep Learning and Practice 1 Chapter 20 Deep Generative Models Peng et al.: Deep Learning and Practice 2 Generative Models Models that are able to Provide an estimate of the probability distribution
More informationDeep Generative Models. (Unsupervised Learning)
Deep Generative Models (Unsupervised Learning) CEng 783 Deep Learning Fall 2017 Emre Akbaş Reminders Next week: project progress demos in class Describe your problem/goal What you have done so far What
More informationSpeaker Representation and Verification Part II. by Vasileios Vasilakakis
Speaker Representation and Verification Part II by Vasileios Vasilakakis Outline -Approaches of Neural Networks in Speaker/Speech Recognition -Feed-Forward Neural Networks -Training with Back-propagation
More informationDeep learning / Ian Goodfellow, Yoshua Bengio and Aaron Courville. - Cambridge, MA ; London, Spis treści
Deep learning / Ian Goodfellow, Yoshua Bengio and Aaron Courville. - Cambridge, MA ; London, 2017 Spis treści Website Acknowledgments Notation xiii xv xix 1 Introduction 1 1.1 Who Should Read This Book?
More informationLarge-Scale Feature Learning with Spike-and-Slab Sparse Coding
Large-Scale Feature Learning with Spike-and-Slab Sparse Coding Ian J. Goodfellow, Aaron Courville, Yoshua Bengio ICML 2012 Presented by Xin Yuan January 17, 2013 1 Outline Contributions Spike-and-Slab
More informationChapter 16. Structured Probabilistic Models for Deep Learning
Peng et al.: Deep Learning and Practice 1 Chapter 16 Structured Probabilistic Models for Deep Learning Peng et al.: Deep Learning and Practice 2 Structured Probabilistic Models way of using graphs to describe
More informationDeep Boltzmann Machines
Deep Boltzmann Machines Ruslan Salakutdinov and Geoffrey E. Hinton Amish Goel University of Illinois Urbana Champaign agoel10@illinois.edu December 2, 2016 Ruslan Salakutdinov and Geoffrey E. Hinton Amish
More informationRepresentational Power of Restricted Boltzmann Machines and Deep Belief Networks. Nicolas Le Roux and Yoshua Bengio Presented by Colin Graber
Representational Power of Restricted Boltzmann Machines and Deep Belief Networks Nicolas Le Roux and Yoshua Bengio Presented by Colin Graber Introduction Representational abilities of functions with some
More informationLearning Deep Architectures
Learning Deep Architectures Yoshua Bengio, U. Montreal Microsoft Cambridge, U.K. July 7th, 2009, Montreal Thanks to: Aaron Courville, Pascal Vincent, Dumitru Erhan, Olivier Delalleau, Olivier Breuleux,
More informationDeep Learning Basics Lecture 8: Autoencoder & DBM. Princeton University COS 495 Instructor: Yingyu Liang
Deep Learning Basics Lecture 8: Autoencoder & DBM Princeton University COS 495 Instructor: Yingyu Liang Autoencoder Autoencoder Neural networks trained to attempt to copy its input to its output Contain
More informationBias-Variance Trade-Off in Hierarchical Probabilistic Models Using Higher-Order Feature Interactions
- Trade-Off in Hierarchical Probabilistic Models Using Higher-Order Feature Interactions Simon Luo The University of Sydney Data61, CSIRO simon.luo@data61.csiro.au Mahito Sugiyama National Institute of
More informationRestricted Boltzmann Machines
Restricted Boltzmann Machines Boltzmann Machine(BM) A Boltzmann machine extends a stochastic Hopfield network to include hidden units. It has binary (0 or 1) visible vector unit x and hidden (latent) vector
More informationCOMP9444 Neural Networks and Deep Learning 11. Boltzmann Machines. COMP9444 c Alan Blair, 2017
COMP9444 Neural Networks and Deep Learning 11. Boltzmann Machines COMP9444 17s2 Boltzmann Machines 1 Outline Content Addressable Memory Hopfield Network Generative Models Boltzmann Machine Restricted Boltzmann
More informationLearning Energy-Based Models of High-Dimensional Data
Learning Energy-Based Models of High-Dimensional Data Geoffrey Hinton Max Welling Yee-Whye Teh Simon Osindero www.cs.toronto.edu/~hinton/energybasedmodelsweb.htm Discovering causal structure as a goal
More informationHow to do backpropagation in a brain
How to do backpropagation in a brain Geoffrey Hinton Canadian Institute for Advanced Research & University of Toronto & Google Inc. Prelude I will start with three slides explaining a popular type of deep
More informationDeep Learning Autoencoder Models
Deep Learning Autoencoder Models Davide Bacciu Dipartimento di Informatica Università di Pisa Intelligent Systems for Pattern Recognition (ISPR) Generative Models Wrap-up Deep Learning Module Lecture Generative
More informationDeep Neural Networks
Deep Neural Networks DT2118 Speech and Speaker Recognition Giampiero Salvi KTH/CSC/TMH giampi@kth.se VT 2015 1 / 45 Outline State-to-Output Probability Model Artificial Neural Networks Perceptron Multi
More informationBasic Principles of Unsupervised and Unsupervised
Basic Principles of Unsupervised and Unsupervised Learning Toward Deep Learning Shun ichi Amari (RIKEN Brain Science Institute) collaborators: R. Karakida, M. Okada (U. Tokyo) Deep Learning Self Organization
More informationRestricted Boltzmann Machines for Collaborative Filtering
Restricted Boltzmann Machines for Collaborative Filtering Authors: Ruslan Salakhutdinov Andriy Mnih Geoffrey Hinton Benjamin Schwehn Presentation by: Ioan Stanculescu 1 Overview The Netflix prize problem
More informationRobust Classification using Boltzmann machines by Vasileios Vasilakakis
Robust Classification using Boltzmann machines by Vasileios Vasilakakis The scope of this report is to propose an architecture of Boltzmann machines that could be used in the context of classification,
More informationPart 2. Representation Learning Algorithms
53 Part 2 Representation Learning Algorithms 54 A neural network = running several logistic regressions at the same time If we feed a vector of inputs through a bunch of logis;c regression func;ons, then
More informationAu-delà de la Machine de Boltzmann Restreinte. Hugo Larochelle University of Toronto
Au-delà de la Machine de Boltzmann Restreinte Hugo Larochelle University of Toronto Introduction Restricted Boltzmann Machines (RBMs) are useful feature extractors They are mostly used to initialize deep
More informationCSC321 Lecture 20: Autoencoders
CSC321 Lecture 20: Autoencoders Roger Grosse Roger Grosse CSC321 Lecture 20: Autoencoders 1 / 16 Overview Latent variable models so far: mixture models Boltzmann machines Both of these involve discrete
More informationDenoising Autoencoders
Denoising Autoencoders Oliver Worm, Daniel Leinfelder 20.11.2013 Oliver Worm, Daniel Leinfelder Denoising Autoencoders 20.11.2013 1 / 11 Introduction Poor initialisation can lead to local minima 1986 -
More informationDeep Learning Architectures and Algorithms
Deep Learning Architectures and Algorithms In-Jung Kim 2016. 12. 2. Agenda Introduction to Deep Learning RBM and Auto-Encoders Convolutional Neural Networks Recurrent Neural Networks Reinforcement Learning
More informationLearning Tetris. 1 Tetris. February 3, 2009
Learning Tetris Matt Zucker Andrew Maas February 3, 2009 1 Tetris The Tetris game has been used as a benchmark for Machine Learning tasks because its large state space (over 2 200 cell configurations are
More informationEmpirical Analysis of the Divergence of Gibbs Sampling Based Learning Algorithms for Restricted Boltzmann Machines
Empirical Analysis of the Divergence of Gibbs Sampling Based Learning Algorithms for Restricted Boltzmann Machines Asja Fischer and Christian Igel Institut für Neuroinformatik Ruhr-Universität Bochum,
More informationDeep Learning & Neural Networks Lecture 2
Deep Learning & Neural Networks Lecture 2 Kevin Duh Graduate School of Information Science Nara Institute of Science and Technology Jan 16, 2014 2/45 Today s Topics 1 General Ideas in Deep Learning Motivation
More informationDoes the Wake-sleep Algorithm Produce Good Density Estimators?
Does the Wake-sleep Algorithm Produce Good Density Estimators? Brendan J. Frey, Geoffrey E. Hinton Peter Dayan Department of Computer Science Department of Brain and Cognitive Sciences University of Toronto
More informationNotes on Boltzmann Machines
1 Notes on Boltzmann Machines Patrick Kenny Centre de recherche informatique de Montréal Patrick.Kenny@crim.ca I. INTRODUCTION Boltzmann machines are probability distributions on high dimensional binary
More informationAn efficient way to learn deep generative models
An efficient way to learn deep generative models Geoffrey Hinton Canadian Institute for Advanced Research & Department of Computer Science University of Toronto Joint work with: Ruslan Salakhutdinov, Yee-Whye
More informationDeep Learning. What Is Deep Learning? The Rise of Deep Learning. Long History (in Hind Sight)
CSCE 636 Neural Networks Instructor: Yoonsuck Choe Deep Learning What Is Deep Learning? Learning higher level abstractions/representations from data. Motivation: how the brain represents sensory information
More informationBayesian Learning in Undirected Graphical Models
Bayesian Learning in Undirected Graphical Models Zoubin Ghahramani Gatsby Computational Neuroscience Unit University College London, UK http://www.gatsby.ucl.ac.uk/ Work with: Iain Murray and Hyun-Chul
More informationAutoencoders and Score Matching. Based Models. Kevin Swersky Marc Aurelio Ranzato David Buchman Benjamin M. Marlin Nando de Freitas
On for Energy Based Models Kevin Swersky Marc Aurelio Ranzato David Buchman Benjamin M. Marlin Nando de Freitas Toronto Machine Learning Group Meeting, 2011 Motivation Models Learning Goal: Unsupervised
More informationLearning Deep Architectures
Learning Deep Architectures Yoshua Bengio, U. Montreal CIFAR NCAP Summer School 2009 August 6th, 2009, Montreal Main reference: Learning Deep Architectures for AI, Y. Bengio, to appear in Foundations and
More informationChapter 11. Stochastic Methods Rooted in Statistical Mechanics
Chapter 11. Stochastic Methods Rooted in Statistical Mechanics Neural Networks and Learning Machines (Haykin) Lecture Notes on Self-learning Neural Algorithms Byoung-Tak Zhang School of Computer Science
More informationNeural Networks. William Cohen [pilfered from: Ziv; Geoff Hinton; Yoshua Bengio; Yann LeCun; Hongkak Lee - NIPs 2010 tutorial ]
Neural Networks William Cohen 10-601 [pilfered from: Ziv; Geoff Hinton; Yoshua Bengio; Yann LeCun; Hongkak Lee - NIPs 2010 tutorial ] WHAT ARE NEURAL NETWORKS? William s notation Logis;c regression + 1
More informationAn Efficient Learning Procedure for Deep Boltzmann Machines Ruslan Salakhutdinov and Geoffrey Hinton
Computer Science and Artificial Intelligence Laboratory Technical Report MIT-CSAIL-TR-2010-037 August 4, 2010 An Efficient Learning Procedure for Deep Boltzmann Machines Ruslan Salakhutdinov and Geoffrey
More informationGaussian Cardinality Restricted Boltzmann Machines
Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence Gaussian Cardinality Restricted Boltzmann Machines Cheng Wan, Xiaoming Jin, Guiguang Ding and Dou Shen School of Software, Tsinghua
More informationRegML 2018 Class 8 Deep learning
RegML 2018 Class 8 Deep learning Lorenzo Rosasco UNIGE-MIT-IIT June 18, 2018 Supervised vs unsupervised learning? So far we have been thinking of learning schemes made in two steps f(x) = w, Φ(x) F, x
More informationWHY ARE DEEP NETS REVERSIBLE: A SIMPLE THEORY,
WHY ARE DEEP NETS REVERSIBLE: A SIMPLE THEORY, WITH IMPLICATIONS FOR TRAINING Sanjeev Arora, Yingyu Liang & Tengyu Ma Department of Computer Science Princeton University Princeton, NJ 08540, USA {arora,yingyul,tengyu}@cs.princeton.edu
More informationA Hybrid Deep Learning Approach For Chaotic Time Series Prediction Based On Unsupervised Feature Learning
A Hybrid Deep Learning Approach For Chaotic Time Series Prediction Based On Unsupervised Feature Learning Norbert Ayine Agana Advisor: Abdollah Homaifar Autonomous Control & Information Technology Institute
More informationEnergy Based Models. Stefano Ermon, Aditya Grover. Stanford University. Lecture 13
Energy Based Models Stefano Ermon, Aditya Grover Stanford University Lecture 13 Stefano Ermon, Aditya Grover (AI Lab) Deep Generative Models Lecture 13 1 / 21 Summary Story so far Representation: Latent
More informationarxiv: v2 [cs.ne] 22 Feb 2013
Sparse Penalty in Deep Belief Networks: Using the Mixed Norm Constraint arxiv:1301.3533v2 [cs.ne] 22 Feb 2013 Xanadu C. Halkias DYNI, LSIS, Universitè du Sud, Avenue de l Université - BP20132, 83957 LA
More informationTraining an RBM: Contrastive Divergence. Sargur N. Srihari
Training an RBM: Contrastive Divergence Sargur N. srihari@cedar.buffalo.edu Topics in Partition Function Definition of Partition Function 1. The log-likelihood gradient 2. Stochastic axiu likelihood and
More informationAn Efficient Learning Procedure for Deep Boltzmann Machines
ARTICLE Communicated by Yoshua Bengio An Efficient Learning Procedure for Deep Boltzmann Machines Ruslan Salakhutdinov rsalakhu@utstat.toronto.edu Department of Statistics, University of Toronto, Toronto,
More informationInductive Principles for Restricted Boltzmann Machine Learning
Inductive Principles for Restricted Boltzmann Machine Learning Benjamin Marlin Department of Computer Science University of British Columbia Joint work with Kevin Swersky, Bo Chen and Nando de Freitas
More informationNeural Networks for Machine Learning. Lecture 11a Hopfield Nets
Neural Networks for Machine Learning Lecture 11a Hopfield Nets Geoffrey Hinton Nitish Srivastava, Kevin Swersky Tijmen Tieleman Abdel-rahman Mohamed Hopfield Nets A Hopfield net is composed of binary threshold
More informationDeep Learning. What Is Deep Learning? The Rise of Deep Learning. Long History (in Hind Sight)
CSCE 636 Neural Networks Instructor: Yoonsuck Choe Deep Learning What Is Deep Learning? Learning higher level abstractions/representations from data. Motivation: how the brain represents sensory information
More informationARestricted Boltzmann machine (RBM) [1] is a probabilistic
1 Matrix Product Operator Restricted Boltzmann Machines Cong Chen, Kim Batselier, Ching-Yun Ko, and Ngai Wong chencong@eee.hku.hk, k.batselier@tudelft.nl, cyko@eee.hku.hk, nwong@eee.hku.hk arxiv:1811.04608v1
More informationTUTORIAL PART 1 Unsupervised Learning
TUTORIAL PART 1 Unsupervised Learning Marc'Aurelio Ranzato Department of Computer Science Univ. of Toronto ranzato@cs.toronto.edu Co-organizers: Honglak Lee, Yoshua Bengio, Geoff Hinton, Yann LeCun, Andrew
More informationLearning to Disentangle Factors of Variation with Manifold Learning
Learning to Disentangle Factors of Variation with Manifold Learning Scott Reed Kihyuk Sohn Yuting Zhang Honglak Lee University of Michigan, Department of Electrical Engineering and Computer Science 08
More informationThe Recurrent Temporal Restricted Boltzmann Machine
The Recurrent Temporal Restricted Boltzmann Machine Ilya Sutskever, Geoffrey Hinton, and Graham Taylor University of Toronto {ilya, hinton, gwtaylor}@cs.utoronto.ca Abstract The Temporal Restricted Boltzmann
More informationIndex. Santanu Pattanayak 2017 S. Pattanayak, Pro Deep Learning with TensorFlow,
Index A Activation functions, neuron/perceptron binary threshold activation function, 102 103 linear activation function, 102 rectified linear unit, 106 sigmoid activation function, 103 104 SoftMax activation
More informationDeep Feedforward Networks
Deep Feedforward Networks Liu Yang March 30, 2017 Liu Yang Short title March 30, 2017 1 / 24 Overview 1 Background A general introduction Example 2 Gradient based learning Cost functions Output Units 3
More informationNonlinear system modeling with deep neural networks and autoencoders algorithm
Nonlinear system modeling with deep neural networks and autoencoders algorithm Erick De la Rosa, Wen Yu Departamento de Control Automatico CINVESTAV-IPN Mexico City, Mexico yuw@ctrl.cinvestav.mx Xiaoou
More informationLearning in Markov Random Fields An Empirical Study
Learning in Markov Random Fields An Empirical Study Sridevi Parise, Max Welling University of California, Irvine sparise,welling@ics.uci.edu Abstract Learning the parameters of an undirected graphical
More informationHopfield Networks and Boltzmann Machines. Christian Borgelt Artificial Neural Networks and Deep Learning 296
Hopfield Networks and Boltzmann Machines Christian Borgelt Artificial Neural Networks and Deep Learning 296 Hopfield Networks A Hopfield network is a neural network with a graph G = (U,C) that satisfies
More informationHow to do backpropagation in a brain. Geoffrey Hinton Canadian Institute for Advanced Research & University of Toronto
1 How to do backpropagation in a brain Geoffrey Hinton Canadian Institute for Advanced Research & University of Toronto What is wrong with back-propagation? It requires labeled training data. (fixed) Almost
More informationCS 6501: Deep Learning for Computer Graphics. Basics of Neural Networks. Connelly Barnes
CS 6501: Deep Learning for Computer Graphics Basics of Neural Networks Connelly Barnes Overview Simple neural networks Perceptron Feedforward neural networks Multilayer perceptron and properties Autoencoders
More informationCourse Structure. Psychology 452 Week 12: Deep Learning. Chapter 8 Discussion. Part I: Deep Learning: What and Why? Rufus. Rufus Processed By Fetch
Psychology 452 Week 12: Deep Learning What Is Deep Learning? Preliminary Ideas (that we already know!) The Restricted Boltzmann Machine (RBM) Many Layers of RBMs Pros and Cons of Deep Learning Course Structure
More informationDeep Belief Networks are Compact Universal Approximators
Deep Belief Networks are Compact Universal Approximators Franck Olivier Ndjakou Njeunje Applied Mathematics and Scientific Computation May 16, 2016 1 / 29 Outline 1 Introduction 2 Preliminaries Universal
More informationUsually the estimation of the partition function is intractable and it becomes exponentially hard when the complexity of the model increases. However,
Odyssey 2012 The Speaker and Language Recognition Workshop 25-28 June 2012, Singapore First attempt of Boltzmann Machines for Speaker Verification Mohammed Senoussaoui 1,2, Najim Dehak 3, Patrick Kenny
More informationArtificial Neural Networks. MGS Lecture 2
Artificial Neural Networks MGS 2018 - Lecture 2 OVERVIEW Biological Neural Networks Cell Topology: Input, Output, and Hidden Layers Functional description Cost functions Training ANNs Back-Propagation
More informationDensity estimation. Computing, and avoiding, partition functions. Iain Murray
Density estimation Computing, and avoiding, partition functions Roadmap: Motivation: density estimation Understanding annealing/tempering NADE Iain Murray School of Informatics, University of Edinburgh
More informationMachine Learning. Neural Networks. (slides from Domingos, Pardo, others)
Machine Learning Neural Networks (slides from Domingos, Pardo, others) Human Brain Neurons Input-Output Transformation Input Spikes Output Spike Spike (= a brief pulse) (Excitatory Post-Synaptic Potential)
More informationBayesian Networks Inference with Probabilistic Graphical Models
4190.408 2016-Spring Bayesian Networks Inference with Probabilistic Graphical Models Byoung-Tak Zhang intelligence Lab Seoul National University 4190.408 Artificial (2016-Spring) 1 Machine Learning? Learning
More informationSTA 4273H: Statistical Machine Learning
STA 4273H: Statistical Machine Learning Russ Salakhutdinov Department of Computer Science! Department of Statistical Sciences! rsalakhu@cs.toronto.edu! h0p://www.cs.utoronto.ca/~rsalakhu/ Lecture 7 Approximate
More informationarxiv: v1 [cs.ne] 6 May 2014
Training Restricted Boltzmann Machine by Perturbation arxiv:1405.1436v1 [cs.ne] 6 May 2014 Siamak Ravanbakhsh, Russell Greiner Department of Computing Science University of Alberta {mravanba,rgreiner@ualberta.ca}
More informationNeural Networks. Advanced data-mining. Yongdai Kim. Department of Statistics, Seoul National University, South Korea
Neural Networks Advanced data-mining Yongdai Kim Department of Statistics, Seoul National University, South Korea What is Neural Networks? One of supervised learning method using one or more hidden layer.
More informationSTA 414/2104: Lecture 8
STA 414/2104: Lecture 8 6-7 March 2017: Continuous Latent Variable Models, Neural networks With thanks to Russ Salakhutdinov, Jimmy Ba and others Outline Continuous latent variable models Background PCA
More informationReplicated Softmax: an Undirected Topic Model. Stephen Turner
Replicated Softmax: an Undirected Topic Model Stephen Turner 1. Introduction 2. Replicated Softmax: A Generative Model of Word Counts 3. Evaluating Replicated Softmax as a Generative Model 4. Experimental
More informationMachine Learning. Neural Networks. (slides from Domingos, Pardo, others)
Machine Learning Neural Networks (slides from Domingos, Pardo, others) For this week, Reading Chapter 4: Neural Networks (Mitchell, 1997) See Canvas For subsequent weeks: Scaling Learning Algorithms toward
More informationStatistical NLP for the Web
Statistical NLP for the Web Neural Networks, Deep Belief Networks Sameer Maskey Week 8, October 24, 2012 *some slides from Andrew Rosenberg Announcements Please ask HW2 related questions in courseworks
More informationRobust Visual Recognition Using Multilayer Generative Neural Networks
Robust Visual Recognition Using Multilayer Generative Neural Networks by Yichuan Tang A thesis presented to the University of Waterloo in fulfillment of the thesis requirement for the degree of Master
More information7.1 Basis for Boltzmann machine. 7. Boltzmann machines
7. Boltzmann machines this section we will become acquainted with classical Boltzmann machines which can be seen obsolete being rarely applied in neurocomputing. It is interesting, after all, because is
More informationArtificial Neural Networks. Introduction to Computational Neuroscience Tambet Matiisen
Artificial Neural Networks Introduction to Computational Neuroscience Tambet Matiisen 2.04.2018 Artificial neural network NB! Inspired by biology, not based on biology! Applications Automatic speech recognition
More informationCheng Soon Ong & Christian Walder. Canberra February June 2018
Cheng Soon Ong & Christian Walder Research Group and College of Engineering and Computer Science Canberra February June 2018 Outlines Overview Introduction Linear Algebra Probability Linear Regression
More informationMixing Rates for the Gibbs Sampler over Restricted Boltzmann Machines
Mixing Rates for the Gibbs Sampler over Restricted Boltzmann Machines Christopher Tosh Department of Computer Science and Engineering University of California, San Diego ctosh@cs.ucsd.edu Abstract The
More informationSTA 414/2104: Lecture 8
STA 414/2104: Lecture 8 6-7 March 2017: Continuous Latent Variable Models, Neural networks Delivered by Mark Ebden With thanks to Russ Salakhutdinov, Jimmy Ba and others Outline Continuous latent variable
More informationAdministration. Registration Hw3 is out. Lecture Captioning (Extra-Credit) Scribing lectures. Questions. Due on Thursday 10/6
Administration Registration Hw3 is out Due on Thursday 10/6 Questions Lecture Captioning (Extra-Credit) Look at Piazza for details Scribing lectures With pay; come talk to me/send email. 1 Projects Projects
More informationMeasuring the Usefulness of Hidden Units in Boltzmann Machines with Mutual Information
Measuring the Usefulness of Hidden Units in Boltzmann Machines with Mutual Information Mathias Berglund, Tapani Raiko, and KyungHyun Cho Department of Information and Computer Science Aalto University
More informationEnhanced Gradient and Adaptive Learning Rate for Training Restricted Boltzmann Machines
Enhanced Gradient and Adaptive Learning Rate for Training Restricted Boltzmann Machines KyungHyun Cho KYUNGHYUN.CHO@AALTO.FI Tapani Raiko TAPANI.RAIKO@AALTO.FI Alexander Ilin ALEXANDER.ILIN@AALTO.FI Department
More informationThe XOR problem. Machine learning for vision. The XOR problem. The XOR problem. x 1 x 2. x 2. x 1. Fall Roland Memisevic
The XOR problem Fall 2013 x 2 Lecture 9, February 25, 2015 x 1 The XOR problem The XOR problem x 1 x 2 x 2 x 1 (picture adapted from Bishop 2006) It s the features, stupid It s the features, stupid The
More informationAn Automated Measure of MDP Similarity for Transfer in Reinforcement Learning
An Automated Measure of MDP Similarity for Transfer in Reinforcement Learning Haitham Bou Ammar University of Pennsylvania Eric Eaton University of Pennsylvania Matthew E. Taylor Washington State University
More informationMixture Models and Representational Power of RBM s, DBN s and DBM s
Mixture Models and Representational Power of RBM s, DBN s and DBM s Guido Montufar Max Planck Institute for Mathematics in the Sciences, Inselstraße 22, D-04103 Leipzig, Germany. montufar@mis.mpg.de Abstract
More information(Feed-Forward) Neural Networks Dr. Hajira Jabeen, Prof. Jens Lehmann
(Feed-Forward) Neural Networks 2016-12-06 Dr. Hajira Jabeen, Prof. Jens Lehmann Outline In the previous lectures we have learned about tensors and factorization methods. RESCAL is a bilinear model for
More informationIntroduction to Artificial Neural Networks
Facultés Universitaires Notre-Dame de la Paix 27 March 2007 Outline 1 Introduction 2 Fundamentals Biological neuron Artificial neuron Artificial Neural Network Outline 3 Single-layer ANN Perceptron Adaline
More information