TTIC 31230, Fundamentals of Deep Learning David McAllester, Winter Variational Autoencoders
|
|
- Corey Barnett
- 5 years ago
- Views:
Transcription
1 TTIC 31230, Fundamentals of Deep Learning David McAllester, Winter 2018 Variational Autoencoders 1
2 The Latent Variable Cross-Entropy Objective We will now drop the negation and switch to argmax. Φ = argmax Φ E y Pop ln Q Φ (y) Q Φ (y) = ẑ Q Φ (ẑ, y) EG Identity: Φ ln Q Φ (y) = Eẑ QΦ (ẑ y) Φ ln Q Φ (ẑ, y) 2
3 Variational Autoencoders Φ ln Q Φ (y) = Eẑ QΦ (ẑ y) Φ ln Q Φ (ẑ, y) Except for directed tree models, this gradient must be approximated exact computation is #-P hard. Variational autoencoders approximate Q Φ (ẑ y) with a model supporting easy sampling of ẑ. 3
4 Generative Models A model for which sampling is easy will be called generative. In Variational autoencoders we assume that Q Φ (y ẑ) is generative but that Q Φ (ẑ y) is not that sampling from Q Φ (ẑ y) is hard. We approximate Q Φ (ẑ y) with a generative model. 4
5 Generation Replaces Search Generation replaces search can be viewed as a general principle of Deep leaning. Rather than search for a ẑ that generates y we strive to directly calculate to generate a ẑ that generates y. Generation replaces search is exemplified in current parsing architectures. 5
6 Variational Autoencoders Φ ln Q Φ (y) = Eẑ QΦ (ẑ y) Φ ln Q Φ (ẑ, y) Φ, Ψ = argmax Φ,Ψ E y Pop Eẑ PΨ (ẑ y) ln Q Φ(ẑ, y) + H(P Ψ (ẑ y)) Here P Ψ (ẑ y) is a generative approximation of Q Φ (ẑ y). The quantity being optimized is called the evidence lower bound (ELBO). 6
7 Variational Autoencoders Φ ln Q Φ (y) = Eẑ QΦ (ẑ y) Φ ln Q Φ (ẑ, y) Φ, Ψ = argmax Φ,Ψ = argmax Φ,Ψ E y Pop Eẑ PΨ (ẑ y) ln Q Φ(ẑ, y) + H(P Ψ (ẑ y)) E y Pop ln Q Φ (y) KL(P Ψ (ẑ y), Q Φ (ẑ y)) The equivalence of the two ELBO expressions is proved below. The first expression supports SGD training through sampling. The second expression establishes that the ELBO is a lower bound on the evidence ln Q Φ (y) and that P Ψ (ẑ y) should approximate Q Φ (ẑ y). 7
8 Derivation of Equivalence I Eẑ PΨ (ẑ y) ln Q Φ(ẑ, y) = Eẑ PΨ (ẑ y) ( ln Q Φ(y) + ln Q Φ (ẑ y) ) = ln Q Φ (y) + Eẑ PΨ (ẑ y) ln Q Φ(ẑ y) = ln Q Φ (y) H(P Ψ (ẑ y), Q Φ (ẑ y)) 8
9 Derivation of Equivalence II Eẑ PΨ (ẑ y) ln Q Φ(ẑ, y) + H(P Ψ (ẑ y)) = ln Q Φ (y) H(P Ψ (ẑ y), Q Φ (ẑ y)) + H(P Ψ (ẑ y)) = ln Q Φ (y) KL(P Ψ (ẑ y), Q Φ (ẑ y)) 9
10 EM is Alternating Optimization of the ELBO Eẑ PΨ (ẑ y) ln Q Φ(ẑ, y) + H(P Ψ (ẑ y)) (1) = ln Q Φ (y) KL(P Ψ (ẑ y), Q Φ (ẑ y)) (2) by (2) Ψ = argmin Ψ E y Pop KL(P Ψ (ẑ y), Q Φ (ẑ y)) by (1) Φ = argmax Φ E y Pop Eẑ PΨ (ẑ y) ln Q Φ(ẑ, y) EM: Φ t+1 = argmax Φ E y Pop Eẑ QΦ t(ẑ y) log Q Φ (ẑ, y) 10
11 The Reparameterization Trick Ψ = argmax Ψ E y Pop Eẑ PΨ (ẑ y) ln Q Φ(ẑ, y) + H(P Ψ (ẑ y)) How do we differentiate the sampling? 11
12 The Reparameterization Trick Ψ = argmax Ψ E y Pop Eẑ PΨ (ẑ y) ln Q Φ(ẑ, y) + H(P Ψ (ẑ y)) We note that in practice all sampling is computed by a deterministic function of (pseudo) random numbers. We can make this explicit. Model P Ψ (ẑ y) by ɛ noise, ẑ = ẑ Ψ (y, ɛ) 12
13 The Reparameterization Trick Ψ = argmax Ψ H(P Ψ (ẑ y)) = E ɛ noise ln P Ψ (ẑ Ψ (y, ɛ) y) E y Pop E ɛ noise ln Q Φ (ẑ Ψ (y, ɛ), y) + H(P Ψ (ẑ y For VAEs we typically we have ẑ(y, ɛ) R d with ẑ(y, ɛ)[i] = µ Ψ (y)[i] + σ Ψ (y)[i] ɛ[i] ɛ[i] N (0, 1) This supports easy calculation of P Ψ (ẑ Ψ (y, ɛ) y). 13
14 Decoding with L 2 Distortion An autoencoder encodes and decodes. We can view ẑ Ψ (y, ɛ) as the encoding of y. We now consider a deterministic decoder ŷ Φ (ẑ) and define a model ( ) y ŷ Q Φ (y ẑ) exp Φ (ẑ) 2 2σ 2
15 A VAE for Images Auto-Encoding Variational Bayes, Diederik P Kingma, Max Welling, y ẑ Ψ (y, ɛ) ẑ ŷ Φ (ẑ) y ŷ 2 [Hyeonwoo Noh et al.] 15
16 Deconvoution: Increasing Spatial Dimension Consider a stride 2 convolution y[i, j, c y ] = W [ i, j, c x, c y ]x[2i + i, 2j + j, c x ] y[i, j, c y ] += B[c y ] For deconvolution we use stride 1 with 4 times the channels. ˆx[i, j, cˆx ] = W [ i, j, cŷ, cˆx ]ŷ[i + i, j + j, cˆx ] ˆx[i, j, cˆx ] += B[cˆx ] The channels at each lower resolution pixel ˆx[i, j] are divided among four higher resolution pixels. This is done by a simple reshaping of ˆx. 16
17 Decoding with L 2 Distortion Φ, Ψ = argmax Φ,Ψ E y Pop Eẑ PΨ (ẑ y) ln Q Φ(ẑ, y) + H(P Ψ (ẑ y)) The objective now becomes E y Pop Eẑ PΨ (ẑ y) ( ln P Φ (ẑ) 1 ) 2σ 2 y ŷ Φ(ẑ) 2 +H(P Ψ (ẑ y)) 17
18 Decoding with L 2 Distortion Switching back to minimization, we can now rewrite the objective as min E y,ɛ ( ẑ Ψ (y, ɛ) Φ λ y ŷ Φ(ẑ Ψ (y, ɛ)) 2 ) ẑ Ψ (y, ɛ) Ψ,y ẑ Φ = log 2 P Φ (ẑ) ẑ Ψ,y = log 2 P Ψ (ẑ y) For ẑ discrete, ẑ Φ is the code length of ẑ(y, ɛ) under an optimal code for P Φ. ẑ Ψ,y is the code length for ẑ under the code for P Ψ (ẑ y). 18
19 Soft EM is to Hard EM as VAE is to Rate-Distortion (Soft) EM: Φ t+1 = argmax Φ E y Pop Eẑ QΦ t(ẑ y) log Q Φ (ẑ, y) Hard EM: Φ t+1 = argmax Φ E y Pop Q Φ (ẑ(y), y) ẑ(y) = argmax ẑ Q Φ t(ẑ y) VAE: min E y,ɛ ẑ Ψ (y, ɛ) Φ λ y ŷ Φ(ẑ Ψ (y, ɛ)) 2 ẑ Ψ (y, ɛ) Ψ,y RD: min E y ẑ Ψ (y) Φ λ y ŷ Φ(ẑ Ψ (y)) 2
20 Sampling P Ψ (ẑ y) ẑ Q Φ (ẑ, y) [Hyeonwoo Noh et al.] Sampling uses just the second half Q Φ (ẑ, y). 20
21 Sampling from Gaussian Variational Autoencoders [Alec Radford] 21
22 Why Blurry? A common explanation for the blurryness of images generated from VAEs is the use of L 2 as the distortion measure. It does seem that L 1 works better (see the slides on image-toimage GANs). However, training on L 2 distortion can produce sharp images in rate-distortion autoencoders (see the slides on ratedistortion autoencoders). 22
23 END
Variational Autoencoders (VAEs)
September 26 & October 3, 2017 Section 1 Preliminaries Kullback-Leibler divergence KL divergence (continuous case) p(x) andq(x) are two density distributions. Then the KL-divergence is defined as Z KL(p
More informationTTIC 31230, Fundamentals of Deep Learning David McAllester, April Information Theory and Distribution Modeling
TTIC 31230, Fundamentals of Deep Learning David McAllester, April 2017 Information Theory and Distribution Modeling Why do we model distributions and conditional distributions using the following objective
More informationVariational Autoencoders
Variational Autoencoders Recap: Story so far A classification MLP actually comprises two components A feature extraction network that converts the inputs into linearly separable features Or nearly linearly
More informationVariational Autoencoder
Variational Autoencoder Göker Erdo gan August 8, 2017 The variational autoencoder (VA) [1] is a nonlinear latent variable model with an efficient gradient-based training procedure based on variational
More informationAuto-Encoding Variational Bayes
Auto-Encoding Variational Bayes Diederik P Kingma, Max Welling June 18, 2018 Diederik P Kingma, Max Welling Auto-Encoding Variational Bayes June 18, 2018 1 / 39 Outline 1 Introduction 2 Variational Lower
More informationTTIC 31230, Fundamentals of Deep Learning David McAllester, Winter Multiclass Logistic Regression. Multilayer Perceptrons (MLPs)
TTIC 31230, Fundamentals of Deep Learning David McAllester, Winter 2018 Multiclass Logistic Regression Multilayer Perceptrons (MLPs) Stochastic Gradient Descent (SGD) 1 Multiclass Classification We consider
More informationExpectation Maximization (EM)
Expectation Maximization (EM) The Expectation Maximization (EM) algorithm is one approach to unsupervised, semi-supervised, or lightly supervised learning. In this kind of learning either no labels are
More informationThe Success of Deep Generative Models
The Success of Deep Generative Models Jakub Tomczak AMLAB, University of Amsterdam CERN, 2018 What is AI about? What is AI about? Decision making: What is AI about? Decision making: new data High probability
More informationarxiv: v1 [cs.lg] 6 Dec 2018
Embedding-reparameterization procedure for manifold-valued latent variables in generative models arxiv:1812.02769v1 [cs.lg] 6 Dec 2018 Eugene Golikov Neural Networks and Deep Learning Lab Moscow Institute
More informationVariational Inference in TensorFlow. Danijar Hafner Stanford CS University College London, Google Brain
Variational Inference in TensorFlow Danijar Hafner Stanford CS 20 2018-02-16 University College London, Google Brain Outline Variational Inference Tensorflow Distributions VAE in TensorFlow Variational
More informationComputing with Distributed Distributional Codes Convergent Inference in Brains and Machines?
Computing with Distributed Distributional Codes Convergent Inference in Brains and Machines? Maneesh Sahani Professor of Theoretical Neuroscience and Machine Learning Gatsby Computational Neuroscience
More informationVariational Auto Encoders
Variational Auto Encoders 1 Recap Deep Neural Models consist of 2 primary components A feature extraction network Where important aspects of the data are amplified and projected into a linearly separable
More informationGenerative Models for Sentences
Generative Models for Sentences Amjad Almahairi PhD student August 16 th 2014 Outline 1. Motivation Language modelling Full Sentence Embeddings 2. Approach Bayesian Networks Variational Autoencoders (VAE)
More informationBayesian Semi-supervised Learning with Deep Generative Models
Bayesian Semi-supervised Learning with Deep Generative Models Jonathan Gordon Department of Engineering Cambridge University jg801@cam.ac.uk José Miguel Hernández-Lobato Department of Engineering Cambridge
More informationVariational Auto-Encoders (VAE)
Variational Auto-Encoders (VAE) Jonathan Pillow Lecture 21 slides NEU 560 Spring 2018 VAE Generative Model latent ~z N (0, I) data f aaacyhicdzhnbhmxemed5assxy0cuvhesfyidiskofb0ungoiktasnk0cpzzxoq/zm+mjsxfeasu8as8ew+dk64qswaks3/n7z9jj2dipfbyfl862z279+4/2huyp3r85omz/ypn5940jkofg2ncymi8skghjwildkwdpiyslibz4xw/widzwugvulqwuuxki1pwhik1qmcvzgdzel9b9ip10f1rtqjl2jgbh3r+vlpdgwuauwted8vc4igwh4jlihnvelcmz9kvdjputiefhfwdi32dmlnag5eorrro/l0rmpj+qsbjqrjo/dzbjf/fhg3wh0zbansgah57ud1iioaupqdt4ycjxcbbubpprztpmgmc0x/leaxhmhulmj6gsuhfdjfhbrnjjkbnywixfrob0xuuhdpegg+bnqnhjusyqjr1d2umhzun/ljwv55tpxjftcqzdkfbehgt6ai4vdkg8xbmy562w27vclech/bkold+ftc9+tjuey+8jk/ig1ks9+sinjaz0iecspknfcc/sk+zza6z5a0167q1l8hgzf9/a5t+5qk=
More informationTTIC 31230, Fundamentals of Deep Learning, Winter David McAllester. The Fundamental Equations of Deep Learning
TTIC 31230, Fundamentals of Deep Learning, Winter 2019 David McAllester The Fundamental Equations of Deep Learning 1 Early History 1943: McCullock and Pitts introduced the linear threshold neuron. 1962:
More informationDeep Variational Inference. FLARE Reading Group Presentation Wesley Tansey 9/28/2016
Deep Variational Inference FLARE Reading Group Presentation Wesley Tansey 9/28/2016 What is Variational Inference? What is Variational Inference? Want to estimate some distribution, p*(x) p*(x) What is
More informationVariational Inference via Stochastic Backpropagation
Variational Inference via Stochastic Backpropagation Kai Fan February 27, 2016 Preliminaries Stochastic Backpropagation Variational Auto-Encoding Related Work Summary Outline Preliminaries Stochastic Backpropagation
More informationGenerative models for missing value completion
Generative models for missing value completion Kousuke Ariga Department of Computer Science and Engineering University of Washington Seattle, WA 98105 koar8470@cs.washington.edu Abstract Deep generative
More informationGENERATIVE ADVERSARIAL LEARNING
GENERATIVE ADVERSARIAL LEARNING OF MARKOV CHAINS Jiaming Song, Shengjia Zhao & Stefano Ermon Computer Science Department Stanford University {tsong,zhaosj12,ermon}@cs.stanford.edu ABSTRACT We investigate
More informationRobust Locally-Linear Controllable Embedding
Ershad Banijamali Rui Shu 2 Mohammad Ghavamzadeh 3 Hung Bui 3 Ali Ghodsi University of Waterloo 2 Standford University 3 DeepMind Abstract Embed-to-control (E2C) 7] is a model for solving high-dimensional
More informationNonparametric Inference for Auto-Encoding Variational Bayes
Nonparametric Inference for Auto-Encoding Variational Bayes Erik Bodin * Iman Malik * Carl Henrik Ek * Neill D. F. Campbell * University of Bristol University of Bath Variational approximations are an
More informationVariational AutoEncoder: An Introduction and Recent Perspectives
Metode de optimizare Riemanniene pentru învățare profundă Proiect cofinanțat din Fondul European de Dezvoltare Regională prin Programul Operațional Competitivitate 2014-2020 Variational AutoEncoder: An
More informationGradient Estimation Using Stochastic Computation Graphs
Gradient Estimation Using Stochastic Computation Graphs Yoonho Lee Department of Computer Science and Engineering Pohang University of Science and Technology May 9, 2017 Outline Stochastic Gradient Estimation
More informationUnsupervised Learning
CS 3750 Advanced Machine Learning hkc6@pitt.edu Unsupervised Learning Data: Just data, no labels Goal: Learn some underlying hidden structure of the data P(, ) P( ) Principle Component Analysis (Dimensionality
More informationDeep latent variable models
Deep latent variable models Pierre-Alexandre Mattei IT University of Copenhagen http://pamattei.github.io @pamattei 19 avril 2018 Séminaire de statistique du CNAM 1 Overview of talk A short introduction
More informationLecture 14: Deep Generative Learning
Generative Modeling CSED703R: Deep Learning for Visual Recognition (2017F) Lecture 14: Deep Generative Learning Density estimation Reconstructing probability density function using samples Bohyung Han
More informationDiscrete Latent Variable Models
Discrete Latent Variable Models Stefano Ermon, Aditya Grover Stanford University Lecture 14 Stefano Ermon, Aditya Grover (AI Lab) Deep Generative Models Lecture 14 1 / 29 Summary Major themes in the course
More informationProbabilistic & Unsupervised Learning
Probabilistic & Unsupervised Learning Week 2: Latent Variable Models Maneesh Sahani maneesh@gatsby.ucl.ac.uk Gatsby Computational Neuroscience Unit, and MSc ML/CSML, Dept Computer Science University College
More informationProbabilistic Graphical Models
10-708 Probabilistic Graphical Models Homework 3 (v1.1.0) Due Apr 14, 7:00 PM Rules: 1. Homework is due on the due date at 7:00 PM. The homework should be submitted via Gradescope. Solution to each problem
More informationBayesian Deep Learning
Bayesian Deep Learning Mohammad Emtiyaz Khan AIP (RIKEN), Tokyo http://emtiyaz.github.io emtiyaz.khan@riken.jp June 06, 2018 Mohammad Emtiyaz Khan 2018 1 What will you learn? Why is Bayesian inference
More informationProbabilistic Graphical Models & Applications
Probabilistic Graphical Models & Applications Learning of Graphical Models Bjoern Andres and Bernt Schiele Max Planck Institute for Informatics The slides of today s lecture are authored by and shown with
More informationAn Information Theoretic Interpretation of Variational Inference based on the MDL Principle and the Bits-Back Coding Scheme
An Information Theoretic Interpretation of Variational Inference based on the MDL Principle and the Bits-Back Coding Scheme Ghassen Jerfel April 2017 As we will see during this talk, the Bayesian and information-theoretic
More informationNatural Gradients via the Variational Predictive Distribution
Natural Gradients via the Variational Predictive Distribution Da Tang Columbia University datang@cs.columbia.edu Rajesh Ranganath New York University rajeshr@cims.nyu.edu Abstract Variational inference
More informationGenerative Adversarial Networks, and Applications
Generative Adversarial Networks, and Applications Ali Mirzaei Nimish Srivastava Kwonjoon Lee Songting Xu CSE 252C 4/12/17 2/44 Outline: Generative Models vs Discriminative Models (Background) Generative
More informationTTIC 31230, Fundamentals of Deep Learning David McAllester, Winter Generalization and Regularization
TTIC 31230, Fundamentals of Deep Learning David McAllester, Winter 2019 Generalization and Regularization 1 Chomsky vs. Kolmogorov and Hinton Noam Chomsky: Natural language grammar cannot be learned by
More informationConvolutional Neural Networks. Srikumar Ramalingam
Convolutional Neural Networks Srikumar Ramalingam Reference Many of the slides are prepared using the following resources: neuralnetworksanddeeplearning.com (mainly Chapter 6) http://cs231n.github.io/convolutional-networks/
More informationA Unified View of Deep Generative Models
SAILING LAB Laboratory for Statistical Artificial InteLigence & INtegreative Genomics A Unified View of Deep Generative Models Zhiting Hu and Eric Xing Petuum Inc. Carnegie Mellon University 1 Deep generative
More informationStochastic Video Prediction with Deep Conditional Generative Models
Stochastic Video Prediction with Deep Conditional Generative Models Rui Shu Stanford University ruishu@stanford.edu Abstract Frame-to-frame stochasticity remains a big challenge for video prediction. The
More informationVariational Dropout via Empirical Bayes
Variational Dropout via Empirical Bayes Valery Kharitonov 1 kharvd@gmail.com Dmitry Molchanov 1, dmolch111@gmail.com Dmitry Vetrov 1, vetrovd@yandex.ru 1 National Research University Higher School of Economics,
More informationSequence labeling. Taking collective a set of interrelated instances x 1,, x T and jointly labeling them
HMM, MEMM and CRF 40-957 Special opics in Artificial Intelligence: Probabilistic Graphical Models Sharif University of echnology Soleymani Spring 2014 Sequence labeling aking collective a set of interrelated
More informationLDA with Amortized Inference
LDA with Amortied Inference Nanbo Sun Abstract This report describes how to frame Latent Dirichlet Allocation LDA as a Variational Auto- Encoder VAE and use the Amortied Variational Inference AVI to optimie
More informationVariational Autoencoders for Classical Spin Models
Variational Autoencoders for Classical Spin Models Benjamin Nosarzewski Stanford University bln@stanford.edu Abstract Lattice spin models are used to study the magnetic behavior of interacting electrons
More informationPrivacy-Preserving Adversarial Networks
MITSUBISHI ELECTRIC RESEARCH LABORATORIES http://www.merl.com Privacy-Preserving Adversarial Networks Tripathy, A.; Wang, Y.; Ishwar, P. TR2017-194 December 2017 Abstract We propose a data-driven framework
More informationEE-559 Deep learning 9. Autoencoders and generative models
EE-559 Deep learning 9. Autoencoders and generative models François Fleuret https://fleuret.org/dlc/ [version of: May 1, 2018] ÉCOLE POLYTECHNIQUE FÉDÉRALE DE LAUSANNE Embeddings and generative models
More informationDeep Generative Models. (Unsupervised Learning)
Deep Generative Models (Unsupervised Learning) CEng 783 Deep Learning Fall 2017 Emre Akbaş Reminders Next week: project progress demos in class Describe your problem/goal What you have done so far What
More informationGenerative Adversarial Networks (GANs) Ian Goodfellow, OpenAI Research Scientist Presentation at Berkeley Artificial Intelligence Lab,
Generative Adversarial Networks (GANs) Ian Goodfellow, OpenAI Research Scientist Presentation at Berkeley Artificial Intelligence Lab, 2016-08-31 Generative Modeling Density estimation Sample generation
More informationCENG 783. Special topics in. Deep Learning. AlchemyAPI. Week 14. Sinan Kalkan
CENG 783 Special topics in Deep Learning AlchemyAPI Week 14 Sinan Kalkan Today Hopfield Networks Boltzmann Machines Deep BM, Restricted BM Generative Adversarial Networks Variational Auto-encoders Autoregressive
More informationarxiv: v1 [stat.ml] 6 Dec 2018
missiwae: Deep Generative Modelling and Imputation of Incomplete Data arxiv:1812.02633v1 [stat.ml] 6 Dec 2018 Pierre-Alexandre Mattei Department of Computer Science IT University of Copenhagen pima@itu.dk
More informationLEARNING TO SAMPLE WITH ADVERSARIALLY LEARNED LIKELIHOOD-RATIO
LEARNING TO SAMPLE WITH ADVERSARIALLY LEARNED LIKELIHOOD-RATIO Chunyuan Li, Jianqiao Li, Guoyin Wang & Lawrence Carin Duke University {chunyuan.li,jianqiao.li,guoyin.wang,lcarin}@duke.edu ABSTRACT We link
More informationConvolution and Pooling as an Infinitely Strong Prior
Convolution and Pooling as an Infinitely Strong Prior Sargur Srihari srihari@buffalo.edu This is part of lecture slides on Deep Learning: http://www.cedar.buffalo.edu/~srihari/cse676 1 Topics in Convolutional
More informationStochastic Backpropagation, Variational Inference, and Semi-Supervised Learning
Stochastic Backpropagation, Variational Inference, and Semi-Supervised Learning Diederik (Durk) Kingma Danilo J. Rezende (*) Max Welling Shakir Mohamed (**) Stochastic Gradient Variational Inference Bayesian
More informationCSC411: Final Review. James Lucas & David Madras. December 3, 2018
CSC411: Final Review James Lucas & David Madras December 3, 2018 Agenda 1. A brief overview 2. Some sample questions Basic ML Terminology The final exam will be on the entire course; however, it will be
More informationDenoising Criterion for Variational Auto-Encoding Framework
Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence (AAAI-17) Denoising Criterion for Variational Auto-Encoding Framework Daniel Jiwoong Im, Sungjin Ahn, Roland Memisevic, Yoshua
More informationAuto-Encoding Variational Bayes. Stochastic Backpropagation and Approximate Inference in Deep Generative Models
Auto-Encoding Variational Bayes Diederik Kingma and Max Welling Stochastic Backpropagation and Approximate Inference in Deep Generative Models Danilo J. Rezende, Shakir Mohamed, Daan Wierstra Neural Variational
More informationDeep Generative Models
Deep Generative Models Durk Kingma Max Welling Deep Probabilistic Models Worksop Wednesday, 1st of Oct, 2014 D.P. Kingma Deep generative models Transformations between Bayes nets and Neural nets Transformation
More informationGenerative adversarial networks
14-1: Generative adversarial networks Prof. J.C. Kao, UCLA Generative adversarial networks Why GANs? GAN intuition GAN equilibrium GAN implementation Practical considerations Much of these notes are based
More informationMACHINE LEARNING AND PATTERN RECOGNITION Fall 2006, Lecture 8: Latent Variables, EM Yann LeCun
Y. LeCun: Machine Learning and Pattern Recognition p. 1/? MACHINE LEARNING AND PATTERN RECOGNITION Fall 2006, Lecture 8: Latent Variables, EM Yann LeCun The Courant Institute, New York University http://yann.lecun.com
More informationConvolutional Neural Network. Hung-yi Lee
al Neural Network Hung-yi Lee Why CNN for Image? [Zeiler, M. D., ECCV 2014] x 1 x 2 Represented as pixels x N The most basic classifiers Use 1 st layer as module to build classifiers Use 2 nd layer as
More informationConvolutional Neural Networks II. Slides from Dr. Vlad Morariu
Convolutional Neural Networks II Slides from Dr. Vlad Morariu 1 Optimization Example of optimization progress while training a neural network. (Loss over mini-batches goes down over time.) 2 Learning rate
More informationSTA 414/2104: Lecture 8
STA 414/2104: Lecture 8 6-7 March 2017: Continuous Latent Variable Models, Neural networks Delivered by Mark Ebden With thanks to Russ Salakhutdinov, Jimmy Ba and others Outline Continuous latent variable
More informationLecture 16 Deep Neural Generative Models
Lecture 16 Deep Neural Generative Models CMSC 35246: Deep Learning Shubhendu Trivedi & Risi Kondor University of Chicago May 22, 2017 Approach so far: We have considered simple models and then constructed
More informationB PROOF OF LEMMA 1. Published as a conference paper at ICLR 2018
A ADVERSARIAL DOMAIN ADAPTATION (ADA ADA aims to transfer prediction nowledge learned from a source domain with labeled data to a target domain without labels, by learning domain-invariant features. Let
More informationBackpropagation Through
Backpropagation Through Backpropagation Through Backpropagation Through Will Grathwohl Dami Choi Yuhuai Wu Geoff Roeder David Duvenaud Where do we see this guy? L( ) =E p(b ) [f(b)] Just about everywhere!
More informationREINTERPRETING IMPORTANCE-WEIGHTED AUTOENCODERS
Worshop trac - ICLR 207 REINTERPRETING IMPORTANCE-WEIGHTED AUTOENCODERS Chris Cremer, Quaid Morris & David Duvenaud Department of Computer Science University of Toronto {ccremer,duvenaud}@cs.toronto.edu
More informationDeep Generative Models for Graph Generation. Jian Tang HEC Montreal CIFAR AI Chair, Mila
Deep Generative Models for Graph Generation Jian Tang HEC Montreal CIFAR AI Chair, Mila Email: jian.tang@hec.ca Deep Generative Models Goal: model data distribution p(x) explicitly or implicitly, where
More informationEnergy-Based Generative Adversarial Network
Energy-Based Generative Adversarial Network Energy-Based Generative Adversarial Network J. Zhao, M. Mathieu and Y. LeCun Learning to Draw Samples: With Application to Amoritized MLE for Generalized Adversarial
More informationCausal Modeling with Generative Neural Networks
Causal Modeling with Generative Neural Networks Michele Sebag TAO, CNRS INRIA LRI Université Paris-Sud Joint work: D. Kalainathan, O. Goudet, I. Guyon, M. Hajaiej, A. Decelle, C. Furtlehner https://arxiv.org/abs/1709.05321
More informationFAST ADAPTATION IN GENERATIVE MODELS WITH GENERATIVE MATCHING NETWORKS
FAST ADAPTATION IN GENERATIVE MODELS WITH GENERATIVE MATCHING NETWORKS Sergey Bartunov & Dmitry P. Vetrov National Research University Higher School of Economics (HSE), Yandex Moscow, Russia ABSTRACT We
More informationCheng Soon Ong & Christian Walder. Canberra February June 2017
Cheng Soon Ong & Christian Walder Research Group and College of Engineering and Computer Science Canberra February June 2017 (Many figures from C. M. Bishop, "Pattern Recognition and ") 1of 679 Part XIX
More informationON THE DIFFERENCE BETWEEN BUILDING AND EX-
ON THE DIFFERENCE BETWEEN BUILDING AND EX- TRACTING PATTERNS: A CAUSAL ANALYSIS OF DEEP GENERATIVE MODELS. Anonymous authors Paper under double-blind review ABSTRACT Generative models are important tools
More informationCitation for published version (APA): Kingma, D. P. (2017). Variational inference & deep learning: A new synthesis
UvA-DARE (Digital Academic Repository) Variational inference & deep learning Kingma, D.P. Link to publication Citation for published version (APA): Kingma, D. P. (2017). Variational inference & deep learning:
More informationVariational Attention for Sequence-to-Sequence Models
Variational Attention for Sequence-to-Sequence Models Hareesh Bahuleyan, 1 Lili Mou, 1 Olga Vechtomova, Pascal Poupart University of Waterloo March, 2018 Outline 1 Introduction 2 Background & Motivation
More informationExpectation Maximization (EM)
Expectation Maximization (EM) The EM algorithm is used to train models involving latent variables using training data in which the latent variables are not observed (unlabeled data). This is to be contrasted
More informationSinging Voice Separation using Generative Adversarial Networks
Singing Voice Separation using Generative Adversarial Networks Hyeong-seok Choi, Kyogu Lee Music and Audio Research Group Graduate School of Convergence Science and Technology Seoul National University
More informationPart 4: Conditional Random Fields
Part 4: Conditional Random Fields Sebastian Nowozin and Christoph H. Lampert Colorado Springs, 25th June 2011 1 / 39 Problem (Probabilistic Learning) Let d(y x) be the (unknown) true conditional distribution.
More informationarxiv: v1 [cs.lg] 15 Jun 2016
Improving Variational Inference with Inverse Autoregressive Flow arxiv:1606.04934v1 [cs.lg] 15 Jun 2016 Diederik P. Kingma, Tim Salimans and Max Welling OpenAI, San Francisco University of Amsterdam, University
More informationNormalized Random Measure Mixture Models in Variational Autoencoders
JMLR: Workshop and Conference Proceedings 1 6 Normalized Random Measure Mixture Models in Variational Autoencoders Rahul Mehta mehta5@uic.edu and Hui Lu huilu@uic.edu Department of Bioengineering University
More informationIntroduction to Machine Learning Midterm Exam Solutions
10-701 Introduction to Machine Learning Midterm Exam Solutions Instructors: Eric Xing, Ziv Bar-Joseph 17 November, 2015 There are 11 questions, for a total of 100 points. This exam is open book, open notes,
More informationAdaGAN: Boosting Generative Models
AdaGAN: Boosting Generative Models Ilya Tolstikhin ilya@tuebingen.mpg.de joint work with Gelly 2, Bousquet 2, Simon-Gabriel 1, Schölkopf 1 1 MPI for Intelligent Systems 2 Google Brain Radford et al., 2015)
More informationPosterior Regularization
Posterior Regularization 1 Introduction One of the key challenges in probabilistic structured learning, is the intractability of the posterior distribution, for fast inference. There are numerous methods
More informationVAE with a VampPrior. Abstract. 1 Introduction
Jakub M. Tomczak University of Amsterdam Max Welling University of Amsterdam Abstract Many different methods to train deep generative models have been introduced in the past. In this paper, we propose
More informationReinforcement Learning
Reinforcement Learning Cyber Rodent Project Some slides from: David Silver, Radford Neal CSC411: Machine Learning and Data Mining, Winter 2017 Michael Guerzhoy 1 Reinforcement Learning Supervised learning:
More informationEM Algorithm LECTURE OUTLINE
EM Algorithm Lukáš Cerman, Václav Hlaváč Czech Technical University, Faculty of Electrical Engineering Department of Cybernetics, Center for Machine Perception 121 35 Praha 2, Karlovo nám. 13, Czech Republic
More informationSTA 414/2104: Lecture 8
STA 414/2104: Lecture 8 6-7 March 2017: Continuous Latent Variable Models, Neural networks With thanks to Russ Salakhutdinov, Jimmy Ba and others Outline Continuous latent variable models Background PCA
More informationCheng Soon Ong & Christian Walder. Canberra February June 2018
Cheng Soon Ong & Christian Walder Research Group and College of Engineering and Computer Science Canberra February June 218 Outlines Overview Introduction Linear Algebra Probability Linear Regression 1
More informationProbabilistic Reasoning in Deep Learning
Probabilistic Reasoning in Deep Learning Dr Konstantina Palla, PhD palla@stats.ox.ac.uk September 2017 Deep Learning Indaba, Johannesburgh Konstantina Palla 1 / 39 OVERVIEW OF THE TALK Basics of Bayesian
More informationarxiv: v3 [stat.ml] 31 Jul 2018
Training VAEs Under Structured Residuals Garoe Dorta 1,2 Sara Vicente 2 Lourdes Agapito 3 Neill D.F. Campbell 1 Ivor Simpson 2 1 University of Bath 2 Anthropics Technology Ltd. 3 University College London
More informationAUTOREGRESSIVE IMPLICIT QUANTILE NETWORKS FOR TIME SERIES GENERATION MAXIMILIEN BAUDRY *
AUTOREGRESSIVE IMPLICIT QUANTILE NETWORKS FOR TIME SERIES GENERATION MAXIMILIEN BAUDRY * *DAMI Chair & Université Lyon 1 - ISFA http://chaire-dami.fr/ AKNOWLEDGEMENTS 2 ACKNOWLEDGEMENTS TO Christian Y.
More informationUnraveling the mysteries of stochastic gradient descent on deep neural networks
Unraveling the mysteries of stochastic gradient descent on deep neural networks Pratik Chaudhari UCLA VISION LAB 1 The question measures disagreement of predictions with ground truth Cat Dog... x = argmin
More informationCheng Soon Ong & Christian Walder. Canberra February June 2018
Cheng Soon Ong & Christian Walder Research Group and College of Engineering and Computer Science Canberra February June 2018 Outlines Overview Introduction Linear Algebra Probability Linear Regression
More informationA Variance Modeling Framework Based on Variational Autoencoders for Speech Enhancement
A Variance Modeling Framework Based on Variational Autoencoders for Speech Enhancement Simon Leglaive 1 Laurent Girin 1,2 Radu Horaud 1 1: Inria Grenoble Rhône-Alpes 2: Univ. Grenoble Alpes, Grenoble INP,
More informationLecture 17: Neural Networks and Deep Learning
UVA CS 6316 / CS 4501-004 Machine Learning Fall 2016 Lecture 17: Neural Networks and Deep Learning Jack Lanchantin Dr. Yanjun Qi 1 Neurons 1-Layer Neural Network Multi-layer Neural Network Loss Functions
More informationSandwiching the marginal likelihood using bidirectional Monte Carlo. Roger Grosse
Sandwiching the marginal likelihood using bidirectional Monte Carlo Roger Grosse Ryan Adams Zoubin Ghahramani Introduction When comparing different statistical models, we d like a quantitative criterion
More informationarxiv: v1 [cs.it] 26 Oct 2018
Outlier Detection using Generative Models with Theoretical Performance Guarantees arxiv:1810.11335v1 [cs.it] 6 Oct 018 Jirong Yi Anh Duc Le Tianming Wang Xiaodong Wu Weiyu Xu October 9, 018 Abstract This
More information1 Machine Learning Concepts (16 points)
CSCI 567 Fall 2018 Midterm Exam DO NOT OPEN EXAM UNTIL INSTRUCTED TO DO SO PLEASE TURN OFF ALL CELL PHONES Problem 1 2 3 4 5 6 Total Max 16 10 16 42 24 12 120 Points Please read the following instructions
More informationBayesian Deep Generative Models for Semi-Supervised and Active Learning
Bayesian Deep Generative Models for Semi-Supervised and Active Learning Jonathan Gordon Supervisor: Dr José Miguel Hernández-Lobato Department of Engineering University of Cambridge This dissertation is
More informationSemi-Supervised Learning with the Deep Rendering Mixture Model
Semi-Supervised Learning with the Deep Rendering Mixture Model Tan Nguyen 1,2 Wanjia Liu 1 Ethan Perez 1 Richard G. Baraniuk 1 Ankit B. Patel 1,2 1 Rice University 2 Baylor College of Medicine 6100 Main
More informationCSCI-567: Machine Learning (Spring 2019)
CSCI-567: Machine Learning (Spring 2019) Prof. Victor Adamchik U of Southern California Mar. 19, 2019 March 19, 2019 1 / 43 Administration March 19, 2019 2 / 43 Administration TA3 is due this week March
More informationIntroduction to Machine Learning Midterm Exam
10-701 Introduction to Machine Learning Midterm Exam Instructors: Eric Xing, Ziv Bar-Joseph 17 November, 2015 There are 11 questions, for a total of 100 points. This exam is open book, open notes, but
More information