The Poisson transform for unnormalised statistical models. Nicolas Chopin (ENSAE) joint work with Simon Barthelmé (CNRS, Gipsa-LAB)
|
|
- June Nelson
- 6 years ago
- Views:
Transcription
1 The Poisson transform for unnormalised statistical models Nicolas Chopin (ENSAE) joint work with Simon Barthelmé (CNRS, Gipsa-LAB)
2 Part I Unnormalised statistical models
3 Unnormalised statistical models Unnormalised statistical models: models with an intractable normalisation constant in the likelihood. Example: Ising model for binary vectors y {0, 1} m p(y a, Q) exp ( ) a t y + y t Qy Very popular in Machine Learning, Computer Vision (deep learning), neuroscience. Creates computational diculties (doubly intractable problems in Bayesian context).
4 Unnormalised sequential models Markov sequence y 0,..., y n where the transition kernel is dened up to a constant. Example: sequential Ising p(y t y t 1, a, Q, R) exp ( a t y t + y t Qy t + y t Ry t 1 ) Nastier than IID version: n normalisation constants missing.
5 Current strategies for inference Classical estimation: MCMC-MLE, contrastive divergence (Bengio and Delalleau, 2009), noise-contrastive divergence (Gutmann and Hyvärinen, 2012). Bayesian: exchange algorithm (Murray et al., 2006), ABC, russian roulette (Girolami et al., 2013). I do not know of methods for sequential unnormalised models.
6 Our contribution Poisson transform shows you can treat the missing normalisation constant as just another parameter. Gives you an alternative likelihood function. Applies to sequential problems as well. Noise-contrastive divergence is an approximation of the Poisson transform and we can now extend it to the sequential setting. Sequential estimation can be turned into a semiparametric logistic regression problem.
7 Part II The Poisson transform
8 Poisson point processes Poisson processes are distributions over countable subsets of a domain Ω (e.g., Ω = R for a temporal point process). Let S be a realisation from a PP. For all (measurable) A Ω, the number of points of S in A follows a Poisson distribution with parameter ˆ λ A = E ( S A ) = λ (y) dy where λ (y) is the intensity function. A
9 Poisson point processes (II) Let's assume that Ω λ (y) dy <, then The cardinal of S is Poisson, with parameter Ω λ (y) dy < ; conditional on S = k, the elements of S are IID with density exp {λ(y)}.
10 Likelihood of a Poisson process log p (S λ) = yi S ˆ log λ (y i ) λ (y) dy Ω
11 The Poisson transform Generalisation of the Poisson-Multinomial transform (Baker, 1994) For estimation purposes, you can treat IID data in just about any space as coming from a Poisson process. New likelihood function: no loss of information, one extra latent parameter.
12 Theorem statement (I) Data: y 1,..., y n Ω, density p(y θ) exp {f θ (y)}, so log-likelihood is L(θ) = n i=1 ˆ f θ (y i ) n log exp {f θ (y)} dy. Ω
13 Theorem statement (I) Data: y 1,..., y n Ω, density p(y θ) exp {f θ (y)}, so log-likelihood is L(θ) = n i=1 ˆ f θ (y i ) n log exp {f θ (y)} dy. Ω Poisson log-likelihood: M (θ, ν) = n ˆ {f θ (y i ) + ν} n exp {f θ (y) + ν} dy Ω i=1 i.e. log-likelihood of a PP with intensity λ(y) = f θ (y) + ν.
14 Theorem statement (II) Theorem ( ) Let θ = argmax L (θ) and θ, ν = argmax M (θ, ν). Then θ Θ θ Θ,ν R θ = θ and ν = log ( exp {f θ (y)} dy). In other words, the MLE can be computed by maximising M (θ, ν) in both variables. There is no loss of information. Also, asymptotic condence intervals for θ are the same. The latent variable ν estimates the normalisation constant.
15 Proof For xed θ, maximise M (θ, ν) wrt ν leads to: (ˆ ) ν (θ) = log exp {f θ (y)} dy and M(θ, ν (θ)) = L(θ) n.
16 Poisson vs. standard likelihood Running example: truncated exponential distribution: y [0, 1], p(y θ) exp (θy)
17 Extension to sequential models The same logic can be applied to sequential models: p θ (y t y t 1 ) exp {f θ (y t, y t 1 )} We will apply the Poisson transform to each conditional distribution.
18 Extension to sequential models Original log-likelihood of sequence: L(θ) = n [ (ˆ )] f θ (y t ; y t 1 ) log exp {f θ (y; y t 1 )} dy Ω t=1
19 Extension to sequential models Original log-likelihood of sequence: L(θ) = n [ (ˆ )] f θ (y t ; y t 1 ) log exp {f θ (y; y t 1 )} dy Ω t=1 Poisson-transformed log-likelihood: M (θ, ν) = n t=1 ˆ {f θ (y t ; y t 1 ) + ν t 1 } Ω n exp {f θ (y; y t 1 ) + ν t 1 } dy We have introduced one latent variable ν t per observation. Sum of integrals becomes integral of a sum. t=1
20 Extension to sequential models Maximising the Poisson-transformed likelihood wrt (θ, ν), gives the MLE for θ, and (ˆ ) ν t 1 (θ ) = log exp {f θ (y; y t 1 )} dy, i.e. minus the log-marginalisation constant for the conditional p(y y t 1, θ ) exp {f θ (y; y t 1 )}.
21 From parametric to semi-parametric inference The value of the latent variables at the mode are a function of y t 1 : (ˆ ) ν t 1 (θ ) = log exp {f θ (y; y t 1 )} dy = χ(y t 1 ). If y t, y t are close, ν t, ν t 1 should be close as well, i.e., χ (y) is (hopefully) smooth.
22 From parametric to semi-parametric inference The value of the latent variables at the mode are a function of y t 1 : (ˆ ) ν t 1 (θ ) = log exp {f θ (y; y t 1 )} dy = χ(y t 1 ). If y t, y t are close, ν t, ν t 1 should be close as well, i.e., χ (y) is (hopefully) smooth. Do inference over χ: e.g. if you have n points but χ is well captured by a spline basis with k n components, use spline basis instead. Poisson likelihood becomes: n M(θ, χ) = {f θ (y t ; y t 1 ) + χ(y t 1 ) } t=1 ˆ Ω t=1 n exp { f θ (y; y t 1 ) + χ(y t 1 ) } dy
23 Using the Poisson transform in practice Back to the IID case: Poisson-transformed likelihood still involves an intractable integral M (θ, ν) = n ˆ {f θ (y i ) + ν} n exp {f θ (y) + ν} dy Ω i=1 which we need to approximate.
24 Using the Poisson transform in practice Back to the IID case: Poisson-transformed likelihood still involves an intractable integral M (θ, ν) = n ˆ {f θ (y i ) + ν} n exp {f θ (y) + ν} dy Ω i=1 which we need to approximate. Several ways, but an interesting one is to go through logistic regression.
25 Stochastic gradient descent Before we go to logistic regression, note that another approach would be to use Monte Carlo (importance sampling) to obtain an unbiased estimate of the gradient: 1 n 1 n θ M (θ, ν) = 1 n n i=1 M (θ, ν) = 1 ν ˆ ˆ θ f (y i; θ) Ω θ f (y i; θ)exp (f θ (y) + ν) dy exp (f θ (y) + ν) dy Ω The we could use SGD (stochastic gradient descent) to maximise M (θ, ν).
26 Part III The logistic trick & noise-contrastive divergence
27 The logistic trick Idea: reduce an estimation problem to a classication problem. Several versions: Logistic regression for density estimation: Hastie et al. (2003), intensity estimation: Baddeley et al. (2010). Logistic regression for normalisation constants: Geyer (1994). Logistic regression for estimation in unnormalised models: Gutmann and Hyvärinen (2012). The last one is called noise-contrastive divergence by the authors.
28 The logistic trick We have n random points from distributions p(y) and n points from q(y). We note z i = 1 if the i-th point is from p, z i = 0 otherwise. Logistic regression models the log-odds ratio: We have that: η(y) = log p(z = 1 y) p(z = 0 y). η(y) = log p(y) q(y) provided q(y) is known, we can rst estimate η (doing some form of logistic regression), and then recover p(y) from η(y).
29 From the logistic trick to noise-contrastive divergence If we have a normalised model p θ (y) then we can run a logistic regression with the following model for the log-odds: η (y; θ) = log p θ (y) log q (y).
30 From the logistic trick to noise-contrastive divergence If we have a normalised model p θ (y) then we can run a logistic regression with the following model for the log-odds: η (y; θ) = log p θ (y) log q (y). If the model is unnormalised, p θ (y) exp {f θ (y)}, we introduce an intercept in the logistic regression η (y; θ) = f θ (y) + ν log q (y). This is the noise-contrastive divergence (NCD) technique of Gutmann and Hyvärinen (2012).
31 Truncated exponential Recall the truncated exponential model: p(y θ) exp (θy) We produce reference samples from U(0, 1), so that the logistic model for NCD is just: Fitting in R: η (y; θ) = θy + ν m <- glm(z~y+offset(logratio),data=df,family=binomial)
32 Summary Logistic trick: get a logistic classier to discriminate true data from random reference data (from a known distribution). It implicitly learns a model for the true data NCD: in unnormalised models, introduce an intercept for the missing normalisation constant Our interpretation: NCD is an approximation of the Poisson-transformed likelihood
33 NCD approximates the Poisson transform In NCD, you can introduce as many reference points (points simulated from q) as you like. Parametrise the log-odds by η (y) = f θ (y) + ν + log n log q(y) m where m is the number of reference points. Theorem: as m +, the logistic log-likelihood R m (θ, ν) tends to the Poisson log-likelihood M (θ, ν) (pointwise).
34 NCD approximates the Poisson transform To sum up: take your true n datapoints, add m random reference datapoints, and estimate the model p θ (y θ) exp {f θ (y)} using a logistic regression with log-odds η (y) = f θ (y) + ν + log n log q(y) m The intercept will be used to estimate the missing normalisation constant. The technique is eectively a practical way of approximating a Poisson-transformed likelihood.
35 NCD for sequential models The relationship between the Poisson transform and NCD shows directly how to adapt NCD to sequential models: apply NCD to each conditional distribution (the transition kernels) Reference density q(y) becomes a reference kernel q (y t y t 1 ) Include an intercept ν t per conditional distribution p(y t y t 1, θ)
36 Truncated exponential, revisited We turn our previous example into a Markov chain: p(y t y t 1, θ) exp (θy t y t 1 ) t y t y t y t 1
37 Truncated exponential, revisited Consider the NCD approximation for xed y t 1. The model for the log-odds will take the form: η (y t ) = θy t y t 1 + ν t 1 + log n m log q(y t y t 1 ) This leads to a linear logistic regression with y t y t 1 as a covariate.
38 Parametric vs. semi-parametric model It is wasteful to t a separate intercept per time-point. As in the semi-parametric version of the Poisson transform, we can use: η (y t ) = θy t y t 1 + χ (y t 1 ) + log n n r log q(y t y t 1 ) where χ (y t 1 ) will be tted using splines.
39 In practice (I) Positive examples are given by: Value at time t 1 Value at time t Label y 1 y 2 1 y 2 y y n 1 y n 1 While negative examples are given by: Value at time t 1 Value at time t Label y 1 r 2 0 y 2 r y n 1 r n 0
40 In practice (II) We can t the (semi-parametric) model via: m <- gam(label ~ I(y_t*y_tminusone)+s(y_tminusone),data= The fully parametric model corresponds to: m <- gam(label ~ I(y_t*y_tminusone)+as.factor(y_tminuson
41 Parametric vs. semi-parametric model Abs. error 10 type NCD semiparametric NCD parametric Maximum likelihood Sample size
42 Part IV Application: LATKES
43 Data Figure : A sequence of eye movements extracted from the dataset of Kienzle et al. (2009). Fixation locations are in red and successive locations are linked by a straight line. Eye movements recorded while 14 subjects where exploring a set of photographs (Fig. 1); each contributing between 600 and 2,000 datapoints.
44 LATKES LATKES: Log-Additive Transition Kernels. A class of spatial Markov chain models, with applications to eye movement data: { } p(y t y t 1,..., y t k ) exp βi v i (y t ) + g(y t, y t 1,..., y t k ) where y 1... y t are spatial locations (e.g. on a screen), v i (y) are spatial covariates, g(...) is an interaction kernel.
45 Fitting LATKES using logistic regression Transition kernel only specied up to normalisation constant. Can use sequential version of NCD to turn the problem into (semiparametric) logistic regression. Standard packages can be used (mgcv, INLA).
46 Example We t the model: p(y t y t 1 ) exp {b( y t ) + r dist ( y t y t 1 ) + r ang ( (y t y t 1 ))} where: b( y t ) should reect a centrality bias; r dist ( y t y t 1 ) should reect the fact that successive xations are close together; r ang ( (y t y t 1 )) should reect a tendency for making movements along the cardinal axes (vertical and horizontal).
47 Note on NCD implementation We tted functions b, r dist and r ang (plus the log-normalising constant χ, as already explained) using smoothing splines. (Extension of NCD to smoothing splines is direct: simply add appropriate penalty to log-likelihood). We used R package mgcv (Wood, 2006). Reference points were sampled from an Uniform distribution (over the screen); 20 reference datapoints per datapoint. Requires one line of code of R, took about 5 minutes.
48 Results Estimated effect π π 2 0 π 2 π Saccade angle (rel. to vertical) Distance to previous fixation (pix.) Distance to center (pix.) Figure : The dierent panels display the estimated eects of saccade angle (r ang ), distance to previous xation (r dist ) and centrality bias (s). Individual subjects are in gray, and the group average is in blue.
49 Conclusion Poisson transform: you can treat any data as coming from a Poisson point process in the appropriate space, and infer the intensity rather than the density. It is OK to treat the normalisation constant as a free parameter! NCD eectively approximates the Poisson transform via logistic regression Inference for unnormalised sequential models can be turned into semi-parametric logistic regression True as well for unnormalised models with covariates See paper on arxiv ( ) and soon in Statistics and Computing.
50 References Baddeley, A., Berman, M., Fisher, N. I., Hardegen, A., Milne, R. K., Schuhmacher, D., Shah, R., and Turner, R. (2010). Spatial logistic regression and change-of-support in poisson point processes. Electronic Journal of Statistics, 4(0): Baker, S. G. (1994). The Multinomial-Poisson transformation. Journal of the Royal Statistical Society. Series D (The Statistician), 43(4): Bengio, Y. and Delalleau, O. (2009). Justifying and generalizing contrastive divergence. Neural computation, 21(6): Geyer, C. J. (1994). Estimating normalizing constants and reweighting mixtures in markov chain monte carlo. Technical Report 568, School of Statistics, University of Minnesota. Girolami, M., Lyne, A.-M., Strathmann, H., Simpson, D., and Atchade, Y. (2013). Playing russian roulette with intractable likelihoods. arxiv Gutmann, M. U. and Hyvärinen, A. (2012). Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics. J. Mach. Learn. Res., 13(1): Hastie, T., Tibshirani, R., and Friedman, J. H. (2003). The Elements of Statistical Learning. Springer, corrected edition. Kienzle, W., Franz, M. O., Schölkopf, B., and Wichmann, F. A. (2009). Center-surround patterns emerge as optimal predictors for human saccade targets. Journal of vision, 9(5).
Nonparametric Bayesian Methods (Gaussian Processes)
[70240413 Statistical Machine Learning, Spring, 2015] Nonparametric Bayesian Methods (Gaussian Processes) Jun Zhu dcszj@mail.tsinghua.edu.cn http://bigml.cs.tsinghua.edu.cn/~jun State Key Lab of Intelligent
More informationBayesian Inference by Density Ratio Estimation
Bayesian Inference by Density Ratio Estimation Michael Gutmann https://sites.google.com/site/michaelgutmann Institute for Adaptive and Neural Computation School of Informatics, University of Edinburgh
More informationEfficient Likelihood-Free Inference
Efficient Likelihood-Free Inference Michael Gutmann http://homepages.inf.ed.ac.uk/mgutmann Institute for Adaptive and Neural Computation School of Informatics, University of Edinburgh 8th November 2017
More informationIntroduction to machine learning and pattern recognition Lecture 2 Coryn Bailer-Jones
Introduction to machine learning and pattern recognition Lecture 2 Coryn Bailer-Jones http://www.mpia.de/homes/calj/mlpr_mpia2008.html 1 1 Last week... supervised and unsupervised methods need adaptive
More informationDistributed Estimation, Information Loss and Exponential Families. Qiang Liu Department of Computer Science Dartmouth College
Distributed Estimation, Information Loss and Exponential Families Qiang Liu Department of Computer Science Dartmouth College Statistical Learning / Estimation Learning generative models from data Topic
More informationMachine Learning. Lecture 4: Regularization and Bayesian Statistics. Feng Li. https://funglee.github.io
Machine Learning Lecture 4: Regularization and Bayesian Statistics Feng Li fli@sdu.edu.cn https://funglee.github.io School of Computer Science and Technology Shandong University Fall 207 Overfitting Problem
More informationSTA414/2104. Lecture 11: Gaussian Processes. Department of Statistics
STA414/2104 Lecture 11: Gaussian Processes Department of Statistics www.utstat.utoronto.ca Delivered by Mark Ebden with thanks to Russ Salakhutdinov Outline Gaussian Processes Exam review Course evaluations
More informationStochastic Backpropagation, Variational Inference, and Semi-Supervised Learning
Stochastic Backpropagation, Variational Inference, and Semi-Supervised Learning Diederik (Durk) Kingma Danilo J. Rezende (*) Max Welling Shakir Mohamed (**) Stochastic Gradient Variational Inference Bayesian
More informationKernel Sequential Monte Carlo
Kernel Sequential Monte Carlo Ingmar Schuster (Paris Dauphine) Heiko Strathmann (University College London) Brooks Paige (Oxford) Dino Sejdinovic (Oxford) * equal contribution April 25, 2016 1 / 37 Section
More informationVariational Inference via Stochastic Backpropagation
Variational Inference via Stochastic Backpropagation Kai Fan February 27, 2016 Preliminaries Stochastic Backpropagation Variational Auto-Encoding Related Work Summary Outline Preliminaries Stochastic Backpropagation
More informationNotes on pseudo-marginal methods, variational Bayes and ABC
Notes on pseudo-marginal methods, variational Bayes and ABC Christian Andersson Naesseth October 3, 2016 The Pseudo-Marginal Framework Assume we are interested in sampling from the posterior distribution
More informationStatistical Data Mining and Machine Learning Hilary Term 2016
Statistical Data Mining and Machine Learning Hilary Term 2016 Dino Sejdinovic Department of Statistics Oxford Slides and other materials available at: http://www.stats.ox.ac.uk/~sejdinov/sdmml Naïve Bayes
More informationTutorial on Approximate Bayesian Computation
Tutorial on Approximate Bayesian Computation Michael Gutmann https://sites.google.com/site/michaelgutmann University of Helsinki Aalto University Helsinki Institute for Information Technology 16 May 2016
More informationBias-Variance Trade-Off in Hierarchical Probabilistic Models Using Higher-Order Feature Interactions
- Trade-Off in Hierarchical Probabilistic Models Using Higher-Order Feature Interactions Simon Luo The University of Sydney Data61, CSIRO simon.luo@data61.csiro.au Mahito Sugiyama National Institute of
More informationBayesian Learning. HT2015: SC4 Statistical Data Mining and Machine Learning. Maximum Likelihood Principle. The Bayesian Learning Framework
HT5: SC4 Statistical Data Mining and Machine Learning Dino Sejdinovic Department of Statistics Oxford http://www.stats.ox.ac.uk/~sejdinov/sdmml.html Maximum Likelihood Principle A generative model for
More informationBayesian Deep Learning
Bayesian Deep Learning Mohammad Emtiyaz Khan AIP (RIKEN), Tokyo http://emtiyaz.github.io emtiyaz.khan@riken.jp June 06, 2018 Mohammad Emtiyaz Khan 2018 1 What will you learn? Why is Bayesian inference
More informationCOMP90051 Statistical Machine Learning
COMP90051 Statistical Machine Learning Semester 2, 2017 Lecturer: Trevor Cohn 17. Bayesian inference; Bayesian regression Training == optimisation (?) Stages of learning & inference: Formulate model Regression
More informationA Review of Pseudo-Marginal Markov Chain Monte Carlo
A Review of Pseudo-Marginal Markov Chain Monte Carlo Discussed by: Yizhe Zhang October 21, 2016 Outline 1 Overview 2 Paper review 3 experiment 4 conclusion Motivation & overview Notation: θ denotes the
More informationLinear Models for Classification
Linear Models for Classification Oliver Schulte - CMPT 726 Bishop PRML Ch. 4 Classification: Hand-written Digit Recognition CHINE INTELLIGENCE, VOL. 24, NO. 24, APRIL 2002 x i = t i = (0, 0, 0, 1, 0, 0,
More informationStein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm
Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm Qiang Liu and Dilin Wang NIPS 2016 Discussion by Yunchen Pu March 17, 2017 March 17, 2017 1 / 8 Introduction Let x R d
More informationEvidence estimation for Markov random fields: a triply intractable problem
Evidence estimation for Markov random fields: a triply intractable problem January 7th, 2014 Markov random fields Interacting objects Markov random fields (MRFs) are used for modelling (often large numbers
More informationSTA 4273H: Statistical Machine Learning
STA 4273H: Statistical Machine Learning Russ Salakhutdinov Department of Computer Science! Department of Statistical Sciences! rsalakhu@cs.toronto.edu! h0p://www.cs.utoronto.ca/~rsalakhu/ Lecture 7 Approximate
More informationSTAT 518 Intro Student Presentation
STAT 518 Intro Student Presentation Wen Wei Loh April 11, 2013 Title of paper Radford M. Neal [1999] Bayesian Statistics, 6: 475-501, 1999 What the paper is about Regression and Classification Flexible
More informationIntractable Likelihood Functions
Intractable Likelihood Functions Michael Gutmann Probabilistic Modelling and Reasoning (INFR11134) School of Informatics, University of Edinburgh Spring semester 2018 Recap p(x y o ) = z p(x,y o,z) x,z
More informationGWAS V: Gaussian processes
GWAS V: Gaussian processes Dr. Oliver Stegle Christoh Lippert Prof. Dr. Karsten Borgwardt Max-Planck-Institutes Tübingen, Germany Tübingen Summer 2011 Oliver Stegle GWAS V: Gaussian processes Summer 2011
More informationProbabilistic Machine Learning. Industrial AI Lab.
Probabilistic Machine Learning Industrial AI Lab. Probabilistic Linear Regression Outline Probabilistic Classification Probabilistic Clustering Probabilistic Dimension Reduction 2 Probabilistic Linear
More informationMonte Carlo in Bayesian Statistics
Monte Carlo in Bayesian Statistics Matthew Thomas SAMBa - University of Bath m.l.thomas@bath.ac.uk December 4, 2014 Matthew Thomas (SAMBa) Monte Carlo in Bayesian Statistics December 4, 2014 1 / 16 Overview
More informationSTA 4273H: Statistical Machine Learning
STA 4273H: Statistical Machine Learning Russ Salakhutdinov Department of Statistics! rsalakhu@utstat.toronto.edu! http://www.utstat.utoronto.ca/~rsalakhu/ Sidney Smith Hall, Room 6002 Lecture 3 Linear
More informationBayesian Methods for Machine Learning
Bayesian Methods for Machine Learning CS 584: Big Data Analytics Material adapted from Radford Neal s tutorial (http://ftp.cs.utoronto.ca/pub/radford/bayes-tut.pdf), Zoubin Ghahramni (http://hunch.net/~coms-4771/zoubin_ghahramani_bayesian_learning.pdf),
More information9/12/17. Types of learning. Modeling data. Supervised learning: Classification. Supervised learning: Regression. Unsupervised learning: Clustering
Types of learning Modeling data Supervised: we know input and targets Goal is to learn a model that, given input data, accurately predicts target data Unsupervised: we know the input only and want to make
More informationIntroduction to Gaussian Process
Introduction to Gaussian Process CS 778 Chris Tensmeyer CS 478 INTRODUCTION 1 What Topic? Machine Learning Regression Bayesian ML Bayesian Regression Bayesian Non-parametric Gaussian Process (GP) GP Regression
More informationProteomics and Variable Selection
Proteomics and Variable Selection p. 1/55 Proteomics and Variable Selection Alex Lewin With thanks to Paul Kirk for some graphs Department of Epidemiology and Biostatistics, School of Public Health, Imperial
More informationA short introduction to INLA and R-INLA
A short introduction to INLA and R-INLA Integrated Nested Laplace Approximation Thomas Opitz, BioSP, INRA Avignon Workshop: Theory and practice of INLA and SPDE November 7, 2018 2/21 Plan for this talk
More informationGatsby Theoretical Neuroscience Lectures: Non-Gaussian statistics and natural images Parts III-IV
Gatsby Theoretical Neuroscience Lectures: Non-Gaussian statistics and natural images Parts III-IV Aapo Hyvärinen Gatsby Unit University College London Part III: Estimation of unnormalized models Often,
More informationEstimating Unnormalized models. Without Numerical Integration
Estimating Unnormalized Models Without Numerical Integration Dept of Computer Science University of Helsinki, Finland with Michael Gutmann Problem: Estimation of unnormalized models We want to estimate
More informationA Magiv CV Theory for Large-Margin Classifiers
A Magiv CV Theory for Large-Margin Classifiers Hui Zou School of Statistics, University of Minnesota June 30, 2018 Joint work with Boxiang Wang Outline 1 Background 2 Magic CV formula 3 Magic support vector
More informationPattern Recognition and Machine Learning
Christopher M. Bishop Pattern Recognition and Machine Learning ÖSpri inger Contents Preface Mathematical notation Contents vii xi xiii 1 Introduction 1 1.1 Example: Polynomial Curve Fitting 4 1.2 Probability
More information> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 BASEL. Logistic Regression. Pattern Recognition 2016 Sandro Schönborn University of Basel
Logistic Regression Pattern Recognition 2016 Sandro Schönborn University of Basel Two Worlds: Probabilistic & Algorithmic We have seen two conceptual approaches to classification: data class density estimation
More informationDensity Estimation. Seungjin Choi
Density Estimation Seungjin Choi Department of Computer Science and Engineering Pohang University of Science and Technology 77 Cheongam-ro, Nam-gu, Pohang 37673, Korea seungjin@postech.ac.kr http://mlg.postech.ac.kr/
More informationNormalising constants and maximum likelihood inference
Normalising constants and maximum likelihood inference Jakob G. Rasmussen Department of Mathematics Aalborg University Denmark March 9, 2011 1/14 Today Normalising constants Approximation of normalising
More informationLatent Variable Models and EM algorithm
Latent Variable Models and EM algorithm SC4/SM4 Data Mining and Machine Learning, Hilary Term 2017 Dino Sejdinovic 3.1 Clustering and Mixture Modelling K-means and hierarchical clustering are non-probabilistic
More informationBANA 7046 Data Mining I Lecture 4. Logistic Regression and Classications 1
BANA 7046 Data Mining I Lecture 4. Logistic Regression and Classications 1 Shaobo Li University of Cincinnati 1 Partially based on Hastie, et al. (2009) ESL, and James, et al. (2013) ISLR Data Mining I
More informationStat 5101 Lecture Notes
Stat 5101 Lecture Notes Charles J. Geyer Copyright 1998, 1999, 2000, 2001 by Charles J. Geyer May 7, 2001 ii Stat 5101 (Geyer) Course Notes Contents 1 Random Variables and Change of Variables 1 1.1 Random
More informationEnergy Based Models. Stefano Ermon, Aditya Grover. Stanford University. Lecture 13
Energy Based Models Stefano Ermon, Aditya Grover Stanford University Lecture 13 Stefano Ermon, Aditya Grover (AI Lab) Deep Generative Models Lecture 13 1 / 21 Summary Story so far Representation: Latent
More informationIntroduction. log p θ (y k y 1:k 1 ), k=1
ESAIM: PROCEEDINGS, September 2007, Vol.19, 115-120 Christophe Andrieu & Dan Crisan, Editors DOI: 10.1051/proc:071915 PARTICLE FILTER-BASED APPROXIMATE MAXIMUM LIKELIHOOD INFERENCE ASYMPTOTICS IN STATE-SPACE
More informationLecture 3. Linear Regression II Bastian Leibe RWTH Aachen
Advanced Machine Learning Lecture 3 Linear Regression II 02.11.2015 Bastian Leibe RWTH Aachen http://www.vision.rwth-aachen.de/ leibe@vision.rwth-aachen.de This Lecture: Advanced Machine Learning Regression
More informationNotes on Noise Contrastive Estimation (NCE)
Notes on Noise Contrastive Estimation NCE) David Meyer dmm@{-4-5.net,uoregon.edu,...} March 0, 207 Introduction In this note we follow the notation used in [2]. Suppose X x, x 2,, x Td ) is a sample of
More informationManaging Uncertainty
Managing Uncertainty Bayesian Linear Regression and Kalman Filter December 4, 2017 Objectives The goal of this lab is multiple: 1. First it is a reminder of some central elementary notions of Bayesian
More informationThe connection of dropout and Bayesian statistics
The connection of dropout and Bayesian statistics Interpretation of dropout as approximate Bayesian modelling of NN http://mlg.eng.cam.ac.uk/yarin/thesis/thesis.pdf Dropout Geoffrey Hinton Google, University
More informationClassification. Chapter Introduction. 6.2 The Bayes classifier
Chapter 6 Classification 6.1 Introduction Often encountered in applications is the situation where the response variable Y takes values in a finite set of labels. For example, the response Y could encode
More informationApproximate Bayesian Computation
Approximate Bayesian Computation Michael Gutmann https://sites.google.com/site/michaelgutmann University of Helsinki and Aalto University 1st December 2015 Content Two parts: 1. The basics of approximate
More informationChapter 4 - Fundamentals of spatial processes Lecture notes
Chapter 4 - Fundamentals of spatial processes Lecture notes Geir Storvik January 21, 2013 STK4150 - Intro 2 Spatial processes Typically correlation between nearby sites Mostly positive correlation Negative
More informationCPSC 540: Machine Learning
CPSC 540: Machine Learning MCMC and Non-Parametric Bayes Mark Schmidt University of British Columbia Winter 2016 Admin I went through project proposals: Some of you got a message on Piazza. No news is
More informationA graph contains a set of nodes (vertices) connected by links (edges or arcs)
BOLTZMANN MACHINES Generative Models Graphical Models A graph contains a set of nodes (vertices) connected by links (edges or arcs) In a probabilistic graphical model, each node represents a random variable,
More informationCSC2535: Computation in Neural Networks Lecture 7: Variational Bayesian Learning & Model Selection
CSC2535: Computation in Neural Networks Lecture 7: Variational Bayesian Learning & Model Selection (non-examinable material) Matthew J. Beal February 27, 2004 www.variational-bayes.org Bayesian Model Selection
More informationLatent Variable Models
Latent Variable Models Stefano Ermon, Aditya Grover Stanford University Lecture 5 Stefano Ermon, Aditya Grover (AI Lab) Deep Generative Models Lecture 5 1 / 31 Recap of last lecture 1 Autoregressive models:
More information10708 Graphical Models: Homework 2
10708 Graphical Models: Homework 2 Due Monday, March 18, beginning of class Feburary 27, 2013 Instructions: There are five questions (one for extra credit) on this assignment. There is a problem involves
More informationGraphical Models and Kernel Methods
Graphical Models and Kernel Methods Jerry Zhu Department of Computer Sciences University of Wisconsin Madison, USA MLSS June 17, 2014 1 / 123 Outline Graphical Models Probabilistic Inference Directed vs.
More informationSummary and discussion of: Dropout Training as Adaptive Regularization
Summary and discussion of: Dropout Training as Adaptive Regularization Statistics Journal Club, 36-825 Kirstin Early and Calvin Murdock November 21, 2014 1 Introduction Multi-layered (i.e. deep) artificial
More informationSTA414/2104 Statistical Methods for Machine Learning II
STA414/2104 Statistical Methods for Machine Learning II Murat A. Erdogdu & David Duvenaud Department of Computer Science Department of Statistical Sciences Lecture 3 Slide credits: Russ Salakhutdinov Announcements
More information13: Variational inference II
10-708: Probabilistic Graphical Models, Spring 2015 13: Variational inference II Lecturer: Eric P. Xing Scribes: Ronghuo Zheng, Zhiting Hu, Yuntian Deng 1 Introduction We started to talk about variational
More informationA Variance Modeling Framework Based on Variational Autoencoders for Speech Enhancement
A Variance Modeling Framework Based on Variational Autoencoders for Speech Enhancement Simon Leglaive 1 Laurent Girin 1,2 Radu Horaud 1 1: Inria Grenoble Rhône-Alpes 2: Univ. Grenoble Alpes, Grenoble INP,
More informationLecture 6: Graphical Models: Learning
Lecture 6: Graphical Models: Learning 4F13: Machine Learning Zoubin Ghahramani and Carl Edward Rasmussen Department of Engineering, University of Cambridge February 3rd, 2010 Ghahramani & Rasmussen (CUED)
More informationLogistic Regression. Machine Learning Fall 2018
Logistic Regression Machine Learning Fall 2018 1 Where are e? We have seen the folloing ideas Linear models Learning as loss minimization Bayesian learning criteria (MAP and MLE estimation) The Naïve Bayes
More informationLecture 5: Linear models for classification. Logistic regression. Gradient Descent. Second-order methods.
Lecture 5: Linear models for classification. Logistic regression. Gradient Descent. Second-order methods. Linear models for classification Logistic regression Gradient descent and second-order methods
More informationCan we do statistical inference in a non-asymptotic way? 1
Can we do statistical inference in a non-asymptotic way? 1 Guang Cheng 2 Statistics@Purdue www.science.purdue.edu/bigdata/ ONR Review Meeting@Duke Oct 11, 2017 1 Acknowledge NSF, ONR and Simons Foundation.
More informationSTA 4273H: Statistical Machine Learning
STA 4273H: Statistical Machine Learning Russ Salakhutdinov Department of Statistics! rsalakhu@utstat.toronto.edu! http://www.utstat.utoronto.ca/~rsalakhu/ Sidney Smith Hall, Room 6002 Lecture 7 Approximate
More informationSTA 414/2104: Machine Learning
STA 414/2104: Machine Learning Russ Salakhutdinov Department of Computer Science! Department of Statistics! rsalakhu@cs.toronto.edu! http://www.cs.toronto.edu/~rsalakhu/ Lecture 9 Sequential Data So far
More informationApproximate Bayesian Computation and Particle Filters
Approximate Bayesian Computation and Particle Filters Dennis Prangle Reading University 5th February 2014 Introduction Talk is mostly a literature review A few comments on my own ongoing research See Jasra
More informationIntroduction to Machine Learning Midterm, Tues April 8
Introduction to Machine Learning 10-701 Midterm, Tues April 8 [1 point] Name: Andrew ID: Instructions: You are allowed a (two-sided) sheet of notes. Exam ends at 2:45pm Take a deep breath and don t spend
More informationIntegrated Likelihood Estimation in Semiparametric Regression Models. Thomas A. Severini Department of Statistics Northwestern University
Integrated Likelihood Estimation in Semiparametric Regression Models Thomas A. Severini Department of Statistics Northwestern University Joint work with Heping He, University of York Introduction Let Y
More informationFast Likelihood-Free Inference via Bayesian Optimization
Fast Likelihood-Free Inference via Bayesian Optimization Michael Gutmann https://sites.google.com/site/michaelgutmann University of Helsinki Aalto University Helsinki Institute for Information Technology
More informationIntroduction to Machine Learning Midterm Exam Solutions
10-701 Introduction to Machine Learning Midterm Exam Solutions Instructors: Eric Xing, Ziv Bar-Joseph 17 November, 2015 There are 11 questions, for a total of 100 points. This exam is open book, open notes,
More informationBased on slides by Richard Zemel
CSC 412/2506 Winter 2018 Probabilistic Learning and Reasoning Lecture 3: Directed Graphical Models and Latent Variables Based on slides by Richard Zemel Learning outcomes What aspects of a model can we
More informationThe Variational Gaussian Approximation Revisited
The Variational Gaussian Approximation Revisited Manfred Opper Cédric Archambeau March 16, 2009 Abstract The variational approximation of posterior distributions by multivariate Gaussians has been much
More informationMachine Learning Basics: Maximum Likelihood Estimation
Machine Learning Basics: Maximum Likelihood Estimation Sargur N. srihari@cedar.buffalo.edu This is part of lecture slides on Deep Learning: http://www.cedar.buffalo.edu/~srihari/cse676 1 Topics 1. Learning
More informationLearning Binary Classifiers for Multi-Class Problem
Research Memorandum No. 1010 September 28, 2006 Learning Binary Classifiers for Multi-Class Problem Shiro Ikeda The Institute of Statistical Mathematics 4-6-7 Minami-Azabu, Minato-ku, Tokyo, 106-8569,
More informationStatistical Methods for Handling Incomplete Data Chapter 2: Likelihood-based approach
Statistical Methods for Handling Incomplete Data Chapter 2: Likelihood-based approach Jae-Kwang Kim Department of Statistics, Iowa State University Outline 1 Introduction 2 Observed likelihood 3 Mean Score
More informationChapter 17: Undirected Graphical Models
Chapter 17: Undirected Graphical Models The Elements of Statistical Learning Biaobin Jiang Department of Biological Sciences Purdue University bjiang@purdue.edu October 30, 2014 Biaobin Jiang (Purdue)
More informationRestricted Boltzmann Machines
Restricted Boltzmann Machines Boltzmann Machine(BM) A Boltzmann machine extends a stochastic Hopfield network to include hidden units. It has binary (0 or 1) visible vector unit x and hidden (latent) vector
More informationGaussian Mixture Models
Gaussian Mixture Models Pradeep Ravikumar Co-instructor: Manuela Veloso Machine Learning 10-701 Some slides courtesy of Eric Xing, Carlos Guestrin (One) bad case for K- means Clusters may overlap Some
More informationLogistic Regression. COMP 527 Danushka Bollegala
Logistic Regression COMP 527 Danushka Bollegala Binary Classification Given an instance x we must classify it to either positive (1) or negative (0) class We can use {1,-1} instead of {1,0} but we will
More informationLecture 2: Simple Classifiers
CSC 412/2506 Winter 2018 Probabilistic Learning and Reasoning Lecture 2: Simple Classifiers Slides based on Rich Zemel s All lecture slides will be available on the course website: www.cs.toronto.edu/~jessebett/csc412
More informationChapter 20. Deep Generative Models
Peng et al.: Deep Learning and Practice 1 Chapter 20 Deep Generative Models Peng et al.: Deep Learning and Practice 2 Generative Models Models that are able to Provide an estimate of the probability distribution
More informationMachine Learning Summer School
Machine Learning Summer School Lecture 3: Learning parameters and structure Zoubin Ghahramani zoubin@eng.cam.ac.uk http://learning.eng.cam.ac.uk/zoubin/ Department of Engineering University of Cambridge,
More informationAn Introduction to Statistical and Probabilistic Linear Models
An Introduction to Statistical and Probabilistic Linear Models Maximilian Mozes Proseminar Data Mining Fakultät für Informatik Technische Universität München June 07, 2017 Introduction In statistical learning
More informationChapter 16. Structured Probabilistic Models for Deep Learning
Peng et al.: Deep Learning and Practice 1 Chapter 16 Structured Probabilistic Models for Deep Learning Peng et al.: Deep Learning and Practice 2 Structured Probabilistic Models way of using graphs to describe
More informationMachine Learning Linear Classification. Prof. Matteo Matteucci
Machine Learning Linear Classification Prof. Matteo Matteucci Recall from the first lecture 2 X R p Regression Y R Continuous Output X R p Y {Ω 0, Ω 1,, Ω K } Classification Discrete Output X R p Y (X)
More informationPh.D. Qualifying Exam Friday Saturday, January 3 4, 2014
Ph.D. Qualifying Exam Friday Saturday, January 3 4, 2014 Put your solution to each problem on a separate sheet of paper. Problem 1. (5166) Assume that two random samples {x i } and {y i } are independently
More informationProbabilistic classification CE-717: Machine Learning Sharif University of Technology. M. Soleymani Fall 2016
Probabilistic classification CE-717: Machine Learning Sharif University of Technology M. Soleymani Fall 2016 Topics Probabilistic approach Bayes decision theory Generative models Gaussian Bayes classifier
More informationMaximum Likelihood, Logistic Regression, and Stochastic Gradient Training
Maximum Likelihood, Logistic Regression, and Stochastic Gradient Training Charles Elkan elkan@cs.ucsd.edu January 17, 2013 1 Principle of maximum likelihood Consider a family of probability distributions
More informationUNIVERSITY of PENNSYLVANIA CIS 520: Machine Learning Final, Fall 2013
UNIVERSITY of PENNSYLVANIA CIS 520: Machine Learning Final, Fall 2013 Exam policy: This exam allows two one-page, two-sided cheat sheets; No other materials. Time: 2 hours. Be sure to write your name and
More informationVariational Autoencoder
Variational Autoencoder Göker Erdo gan August 8, 2017 The variational autoencoder (VA) [1] is a nonlinear latent variable model with an efficient gradient-based training procedure based on variational
More informationMachine Learning - MT & 5. Basis Expansion, Regularization, Validation
Machine Learning - MT 2016 4 & 5. Basis Expansion, Regularization, Validation Varun Kanade University of Oxford October 19 & 24, 2016 Outline Basis function expansion to capture non-linear relationships
More informationOutline Lecture 2 2(32)
Outline Lecture (3), Lecture Linear Regression and Classification it is our firm belief that an understanding of linear models is essential for understanding nonlinear ones Thomas Schön Division of Automatic
More informationApproximate Bayesian inference
Approximate Bayesian inference Variational and Monte Carlo methods Christian A. Naesseth 1 Exchange rate data 0 20 40 60 80 100 120 Month Image data 2 1 Bayesian inference 2 Variational inference 3 Stochastic
More informationPart 4: Conditional Random Fields
Part 4: Conditional Random Fields Sebastian Nowozin and Christoph H. Lampert Colorado Springs, 25th June 2011 1 / 39 Problem (Probabilistic Learning) Let d(y x) be the (unknown) true conditional distribution.
More informationProbabilistic Graphical Models Lecture 17: Markov chain Monte Carlo
Probabilistic Graphical Models Lecture 17: Markov chain Monte Carlo Andrew Gordon Wilson www.cs.cmu.edu/~andrewgw Carnegie Mellon University March 18, 2015 1 / 45 Resources and Attribution Image credits,
More informationAdaptive HMC via the Infinite Exponential Family
Adaptive HMC via the Infinite Exponential Family Arthur Gretton Gatsby Unit, CSML, University College London RegML, 2017 Arthur Gretton (Gatsby Unit, UCL) Adaptive HMC via the Infinite Exponential Family
More informationLink lecture - Lagrange Multipliers
Link lecture - Lagrange Multipliers Lagrange multipliers provide a method for finding a stationary point of a function, say f(x, y) when the variables are subject to constraints, say of the form g(x, y)
More information