Latent state estimation using control theory
|
|
- Silvia Beasley
- 5 years ago
- Views:
Transcription
1 Latent state estimation using control theory Bert Kappen SNN Donders Institute, Radboud University, Nijmegen Gatsby Unit, UCL London August 3, 7 with Hans Christian Ruiz Bert Kappen
2 Smoothing problem Given observed data y :T, estimate (statistics of) marginal posteriors p(x t y :T ) with p(x :T y :T ) = p(x :T)p(y :T x :T ) p(y :T ) p(y :T ) = x :T p(x :T )p(y :T x :T ) NB is different from filtering problem p(x t y :t ). Bert Kappen
3 Smoothing problem Given observed data y :T, estimate (statistics of) marginal posteriors p(x t y :T ) with p(x :T y :T ) = p(x :T)p(y :T x :T ) p(y :T ) p(y :T ) = x :T p(x :T )p(y :T x :T ) NB is different from filtering problem p(x t y :t ). Common approaches: Kalman Filtering: exact for linear dynamics and Gaussian observations Variational, EP, extended Kalman Filter: valid for simple posterior distributions Particle filtering, SMC: generic, but inefficient Bert Kappen
4 Time series inference Prior process p(x :T x ): dx t = dw t x = Observation at end time only: p(y T x T ) = exp( βx T ) t Red: p(x t y :t, x ) (exact) Blue: p(x t y :t, x ) (PF) Black: p(x t y :T, x ) (exact) Bert Kappen 3
5 Time series inference Prior process p(x :T x ): dx t = dw t x = Observation at end time only: p(y T x T ) = exp( βx T ) t Red: p(x t y :t, x ) (exact) Blue: p(x t y :t, x ) (PF + resampling) Black: p(x t y :T, x ) (exact) Bert Kappen 4
6 Time series inference Prior process p(x :T x ): dx t = dw t x = Observation at end time only: p(y T x T ) = exp( βx T ) t t Samples from filtered distribution do not well represent smoothed distribution. Bert Kappen 5
7 Control approach to time-series inference Outline: Introduction to control theory and path integral control theory Optimal control optimal sampling Adaptive importance sampler: Estimate improved sampler from self-generated data fmri application Bert Kappen 6
8 Control theory.5.5 Consider a stochastic dynamical system dx t = f (X t, u)dt + dw t E(dW t,i dw t. j ) = ν i j dt Given X find control function u(x, t) that minimizes the expected future cost C = E ( φ(x T ) + T ) dtv(x t, u(x t, t)) Bert Kappen 7
9 Control theory Standard approach: define J(x, t) is optimal cost-to-go from x, t. J(x, t) = min u t:t E u (φ(x T ) + T t ) dtv(x t, u(x t, t)) X t = x J satisfies a partial differential equation t J(t, x) = min u ( V(x, u) + f (x, u) x J(x, t) + ) ν xj(x, t) J(x, T) = φ(x) with u = u(x, t).this is HJB equation. Optimal control u (x, t) defines distribution over trajectories p (τ) (= p(τ x, )). Bert Kappen 8
10 Path integral control theory.5.5 dx t = f (X t, t)dt + g(x t, t)(u(x t, t)dt + dw t ) X = x Goal is to find function u(x, t) that minimizes C = E u ( S (τ) + T dt u(x t, t) ) S (τ) = φ(x T ) + T V(X t, t) Bert Kappen 9
11 Path integral control theory Equivalent formulation: Find distribution over trajectories p that minimizes C(p) = dτp(τ) ( S (τ) + log p(τ) ) q(τ) q(τ x, ) is distribution over uncontrolled trajectories. The optimal solution is given by p (τ) = (τ) ψq(τ)e S E u T dt u(x t, t) = dτp(τ) log p(τ) q(τ). Bert Kappen
12 Path integral control theory Equivalent formulation: Find distribution over trajectories p that minimizes C(p) = dτp(τ) ( S (τ) + log p(τ) ) q(τ) q(τ x, ) is distribution over uncontrolled trajectories. The optimal solution is given by p (τ) = ψ q(τ)e S (τ) = p(τ u ). Equivalence of optimal control and discounted cost (Girsanov) Bert Kappen
13 Path integral control theory The optimal control cost is C(p ) = log ψ = J(x, ) with ψ = dτq(τ)e S (τ) = E q e S J(x, t) can be computed by forward sampling from q. Bert Kappen
14 Estimating ψ = Ee S ESS =.8, C= Sample N trajectories from uncontrolled dynamics τ i q(τ) w i = e S (τ i) ˆψ = N i w i ˆψ unbiased estimate of ψ. Sampling efficiency is inversely proportional to variance in (normalized) w i. ES S = N + N Var(w) Bert Kappen 3
15 Importance sampling ESS =.8, C=3.7 ESS = 3.5, C=5. ESS=9.5, C= Sample N trajectories from controlled dynamics and reweight yields unbiased estimate of cost-to-go: τ i p(τ) w i = e S (τ i) q(τ i) p(τ i ) = e S u(τ i ) ˆψ = N i w i S u (τ) = S (τ) + T dt u(x t, t) + T u(x t, t)dw t Bert Kappen 4
16 Importance sampling ESS =.8, C=3.7 ESS = 3.5, C=5. ESS=9.5, C= S u (τ) = S (τ) + T dt u(x t, t) + T u(x t, t)dw t Thm: Better u (in the sense of optimal control) provides a better sampler (in the sense of effective sample size). Optimal u = u (in the sense of optimal control) requires only one sample and S u (τ) deterministic! Thijssen, Kappen 5 Bert Kappen 5
17 Proof Control cost is C(p) = E p ( S (τ) + log p(τ) q(τ)) = ES u Using Jensen s inequality: C = log q(τ)e S (τ) = log τ τ p(τ) S (τ) log p(τ)e q(τ) τ p(τ) ( S (τ) + log p(τ) ) q(τ) Bert Kappen 6
18 ( ) Control cost is C(p) = E p S (τ) + log p(τ) q(τ) Proof Using Jensen s inequality: C = log q(τ)e S (τ) = log τ τ p(τ) S (τ) log p(τ)e q(τ) τ p(τ) ( S (τ) + log p(τ) ) q(τ) = C(p) The inequality is saturated when S (τ) + log p(τ) q(τ) side evaluate to S (τ) + log p(τ) q(τ). has zero variance: left and right This is realized when p = p 3. 3 p exists when τ q(τ)e S (τ) < Bert Kappen 7
19 The Path Integral Cross Entropy (PICE) method We wish to estimate ψ = S (τ) dτq(τ)e The optimal (zero variance) importance sampler is p (τ) = ψ q(τ)e S (τ). We approximate p (τ) with p u (τ), where u(x, t θ) is a parametrized control function. Following the Cross Entropy method, we minimise KL(p p u ). θ KL(p p u ) θ E u e S u T dw t u(x t, t θ) θ u(x, t θ) is arbitrary. Estimate gradient by sampling. Kappen, Ruiz 6 Bert Kappen 8
20 Coordination of UAVs Chao Xu ACC 7 Bert Kappen 9
21 Control Inference Fleming, Mitter Bert Kappen
22 Path integral control theory Equivalent formulation: Find distribution over trajectories p that minimizes C(p) = dτp(τ) ( S (τ) + log p(τ) ) q(τ) q(τ x, ) is distribution over uncontrolled trajectories. The optimal solution is given by p (τ) = ψ q(τ)e S (τ) = p(τ u ). q(τ) is the prior process p(x :T x ) and e S (τ) = t p(y t x t ). Bert Kappen
23 Control versus particle smoothing Linear stochastic dynamics, Gaussian observations at t =, T. Control method with linear time dependent controller u(x, t) = a(t)x+b(t). N = particles, 5 IS iterations. ESS from.5% to 98%. FS with N = particles. ESS = %. FFBSi with N = M = forward and backward particles. ESS = 7%. Control error in posterior mean is orders of magnitude smaller than FFBSi error. Bert Kappen
24 Control versus particle smoothing Linear stochastic dynamics, Gaussian observations at t =, T. FFBSi error in posterior mean (average over time) increases with more unlikely observations, while control error does not. Bert Kappen 3
25 Control method scales with many observations Linear stochastic dynamics, Gaussian observations at t =, T. Bert Kappen 4
26 Non linear problem 5 dimensional neural network model with 4 observation of one neuron. Left: Number of iterations needed to reach ESS=. versus number of observations (N = 6). Annealing vital for large number of observations. Right: Number of iterations needed to reach ESS=. versus number of particles (5 observations). Annealing allows significant less number of particles. Bert Kappen 5
27 Non linear problem 5 dimensional neural network model with 4 observation of one neuron. Variance of posterior mean for APIS, FS and FFBSi versus number of observations. N = 6 forward particles. Backward particles such that APIS and FFBSi have similar CPU time. Bert Kappen 6
28 Non linear problem 5 dimensional neural network model with 4 observation of one neuron. ESS for typical run of APIS with annealing. N = 75 Bert Kappen 7
29 Infering neural activity from BOLD fmri Subjects were asked to respond as fast as possible to a visual or auditory stimulus Measure fmri BOLD response in motor cortex Reconstruct neural signal from BOLD signal. Neural activity dz t = AZ t dt + σ z dw t Hemodynamics Vasodilation s(z, s, f ) Blood flow f (s) Balloon model Blood volume v( f, v) Deoxygenated hemoglobin q( f, v, q) BOLD y(v, q) Bert Kappen 8
30 Infering neural activity from BOLD fmri Subjects were asked to respond as fast as possible to a visual or auditory stimulus Measure fmri BOLD response in motor cortex Reconstruct neural signal from BOLD signal. Neural activity dz t = AZ t dt + σ z dw t Hemodynamics Vasodilation s(z, s, f ) Blood flow f (s) Balloon model Blood volume v( f, v) Deoxygenated hemoglobin q( f, v, q) BOLD y(v, q) Bert Kappen 9
31 Infering neural activity from BOLD fmri Subjects were asked to respond as fast as possible to a visual or auditory stimulus Measure fmri BOLD response in motor cortex Reconstruct neural signal from BOLD signal. Neural activity dz t = AZ t dt + σ z dw t Hemodynamics Vasodilation s(z, s, f ) Blood flow f (s) Balloon model Blood volume v( f, v) Deoxygenated hemoglobin q( f, v, q) BOLD y(v, q) Bert Kappen 3
32 Infering neural activity from BOLD fmri u(x, t) = a(t)x + b(t). 5. particles, iterations. Error in stimulus time reconstruction.5.77s. Error using fit of two Gamma functions.8.5s Bert Kappen 3
33 Summary Smoothing problem (Bayesian posterior computation in time series) can be formulated as a stochastic optimal control problem (Fleming 99). Bert Kappen 3
34 Summary Smoothing problem (Bayesian posterior computation in time series) can be formulated as a stochastic optimal control problem (Fleming 99). Better controller = Better sampler. Bert Kappen 33
35 Summary Smoothing problem (Bayesian posterior computation in time series) can be formulated as a stochastic optimal control problem (Fleming 99). Better controller = Better sampler. Parametrized controller found by optimizing Cross Entropy can be (in principle) arbitrary accurate. Bert Kappen 34
36 Summary Smoothing problem (Bayesian posterior computation in time series) can be formulated as a stochastic optimal control problem (Fleming 99). Better controller = Better sampler. Parametrized controller found by optimizing Cross Entropy can be (in principle) arbitrary accurate. Improved performance compared to state of the art SMC. Bert Kappen 35
37 Summary Smoothing problem (Bayesian posterior computation in time series) can be formulated as a stochastic optimal control problem (Fleming 99). Better controller = Better sampler. Parametrized controller found by optimizing Cross Entropy can be (in principle) arbitrary accurate. Improved performance compared to state of the art SMC. Excellent parallelization possible. Bert Kappen 36
38 Thank you! Kappen, Hilbert Johan, and Hans Christian Ruiz. Adaptive importance sampling for control and inference. Journal of Statistical Physics 6.5 (6): Ruiz, Hans-Christian, and Hilbert J. Kappen. Particle Smoothing for Hidden Diffusion Processes: Adaptive Path Integral Smoother. IEEE Transactions on Signal Processing 65. (7): Bert Kappen 37
An efficient approach to stochastic optimal control. Bert Kappen SNN Radboud University Nijmegen the Netherlands
An efficient approach to stochastic optimal control Bert Kappen SNN Radboud University Nijmegen the Netherlands Bert Kappen Examples of control tasks Motor control Bert Kappen Pascal workshop, 27-29 May
More informationSmoothing Estimates of Diffusion Processes
Smoothing Estimates of Diffusion Processes H.C. Ruiz Euler Donders Institute Radboud University Nijmegen, NL hruiz@science.ru.nl H.J. Kappen Donders Institute Radboud University Nijmegen, NL bkappen@science.ru.nl
More informationStochastic optimal control theory
Stochastic optimal control theory Bert Kappen SNN Radboud University Nijmegen the Netherlands July 5, 2008 Bert Kappen Introduction Optimal control theory: Optimize sum of a path cost and end cost. Result
More informationControlled sequential Monte Carlo
Controlled sequential Monte Carlo Jeremy Heng, Department of Statistics, Harvard University Joint work with Adrian Bishop (UTS, CSIRO), George Deligiannidis & Arnaud Doucet (Oxford) Bayesian Computation
More informationInformation geometry for bivariate distribution control
Information geometry for bivariate distribution control C.T.J.Dodson + Hong Wang Mathematics + Control Systems Centre, University of Manchester Institute of Science and Technology Optimal control of stochastic
More informationApproximate Bayesian inference
Approximate Bayesian inference Variational and Monte Carlo methods Christian A. Naesseth 1 Exchange rate data 0 20 40 60 80 100 120 Month Image data 2 1 Bayesian inference 2 Variational inference 3 Stochastic
More informationSequential Monte Carlo Methods for Bayesian Computation
Sequential Monte Carlo Methods for Bayesian Computation A. Doucet Kyoto Sept. 2012 A. Doucet (MLSS Sept. 2012) Sept. 2012 1 / 136 Motivating Example 1: Generic Bayesian Model Let X be a vector parameter
More informationExpectation Propagation in Dynamical Systems
Expectation Propagation in Dynamical Systems Marc Peter Deisenroth Joint Work with Shakir Mohamed (UBC) August 10, 2012 Marc Deisenroth (TU Darmstadt) EP in Dynamical Systems 1 Motivation Figure : Complex
More informationPath Integral Stochastic Optimal Control for Reinforcement Learning
Preprint August 3, 204 The st Multidisciplinary Conference on Reinforcement Learning and Decision Making RLDM203 Path Integral Stochastic Optimal Control for Reinforcement Learning Farbod Farshidian Institute
More informationMarkov Chain Monte Carlo Methods for Stochastic
Markov Chain Monte Carlo Methods for Stochastic Optimization i John R. Birge The University of Chicago Booth School of Business Joint work with Nicholas Polson, Chicago Booth. JRBirge U Florida, Nov 2013
More informationProbabilistic Graphical Models for Image Analysis - Lecture 4
Probabilistic Graphical Models for Image Analysis - Lecture 4 Stefan Bauer 12 October 2018 Max Planck ETH Center for Learning Systems Overview 1. Repetition 2. α-divergence 3. Variational Inference 4.
More informationFully Bayesian Deep Gaussian Processes for Uncertainty Quantification
Fully Bayesian Deep Gaussian Processes for Uncertainty Quantification N. Zabaras 1 S. Atkinson 1 Center for Informatics and Computational Science Department of Aerospace and Mechanical Engineering University
More informationLecture 6: Bayesian Inference in SDE Models
Lecture 6: Bayesian Inference in SDE Models Bayesian Filtering and Smoothing Point of View Simo Särkkä Aalto University Simo Särkkä (Aalto) Lecture 6: Bayesian Inference in SDEs 1 / 45 Contents 1 SDEs
More informationBasic math for biology
Basic math for biology Lei Li Florida State University, Feb 6, 2002 The EM algorithm: setup Parametric models: {P θ }. Data: full data (Y, X); partial data Y. Missing data: X. Likelihood and maximum likelihood
More informationHierarchy. Will Penny. 24th March Hierarchy. Will Penny. Linear Models. Convergence. Nonlinear Models. References
24th March 2011 Update Hierarchical Model Rao and Ballard (1999) presented a hierarchical model of visual cortex to show how classical and extra-classical Receptive Field (RF) effects could be explained
More informationComputer Intensive Methods in Mathematical Statistics
Computer Intensive Methods in Mathematical Statistics Department of mathematics johawes@kth.se Lecture 16 Advanced topics in computational statistics 18 May 2017 Computer Intensive Methods (1) Plan of
More informationProbabilistic Graphical Models
Probabilistic Graphical Models Brown University CSCI 2950-P, Spring 2013 Prof. Erik Sudderth Lecture 12: Gaussian Belief Propagation, State Space Models and Kalman Filters Guest Kalman Filter Lecture by
More informationLecture 1a: Basic Concepts and Recaps
Lecture 1a: Basic Concepts and Recaps Cédric Archambeau Centre for Computational Statistics and Machine Learning Department of Computer Science University College London c.archambeau@cs.ucl.ac.uk Advanced
More informationGaussian Process Approximations of Stochastic Differential Equations
Gaussian Process Approximations of Stochastic Differential Equations Cédric Archambeau Dan Cawford Manfred Opper John Shawe-Taylor May, 2006 1 Introduction Some of the most complex models routinely run
More informationPolicy Search for Path Integral Control
Policy Search for Path Integral Control Vicenç Gómez 1,2, Hilbert J Kappen 2, Jan Peters 3,4, and Gerhard Neumann 3 1 Universitat Pompeu Fabra, Barcelona Department of Information and Communication Technologies,
More informationMarkov Chain Monte Carlo Methods for Stochastic Optimization
Markov Chain Monte Carlo Methods for Stochastic Optimization John R. Birge The University of Chicago Booth School of Business Joint work with Nicholas Polson, Chicago Booth. JRBirge U of Toronto, MIE,
More informationSequential Monte Carlo Samplers for Applications in High Dimensions
Sequential Monte Carlo Samplers for Applications in High Dimensions Alexandros Beskos National University of Singapore KAUST, 26th February 2014 Joint work with: Dan Crisan, Ajay Jasra, Nik Kantas, Alex
More informationParticle Filtering Approaches for Dynamic Stochastic Optimization
Particle Filtering Approaches for Dynamic Stochastic Optimization John R. Birge The University of Chicago Booth School of Business Joint work with Nicholas Polson, Chicago Booth. JRBirge I-Sim Workshop,
More informationIntegrated Non-Factorized Variational Inference
Integrated Non-Factorized Variational Inference Shaobo Han, Xuejun Liao and Lawrence Carin Duke University February 27, 2014 S. Han et al. Integrated Non-Factorized Variational Inference February 27, 2014
More informationLinear Dynamical Systems
Linear Dynamical Systems Sargur N. srihari@cedar.buffalo.edu Machine Learning Course: http://www.cedar.buffalo.edu/~srihari/cse574/index.html Two Models Described by Same Graph Latent variables Observations
More informationAuxiliary Particle Methods
Auxiliary Particle Methods Perspectives & Applications Adam M. Johansen 1 adam.johansen@bristol.ac.uk Oxford University Man Institute 29th May 2008 1 Collaborators include: Arnaud Doucet, Nick Whiteley
More informationState-Space Methods for Inferring Spike Trains from Calcium Imaging
State-Space Methods for Inferring Spike Trains from Calcium Imaging Joshua Vogelstein Johns Hopkins April 23, 2009 Joshua Vogelstein (Johns Hopkins) State-Space Calcium Imaging April 23, 2009 1 / 78 Outline
More informationEffective Connectivity & Dynamic Causal Modelling
Effective Connectivity & Dynamic Causal Modelling Hanneke den Ouden Donders Centre for Cognitive Neuroimaging Radboud University Nijmegen Advanced SPM course Zurich, Februari 13-14, 2014 Functional Specialisation
More informationStatistical Inference and Methods
Department of Mathematics Imperial College London d.stephens@imperial.ac.uk http://stats.ma.ic.ac.uk/ das01/ 31st January 2006 Part VI Session 6: Filtering and Time to Event Data Session 6: Filtering and
More informationProbabilistic and Bayesian Machine Learning
Probabilistic and Bayesian Machine Learning Day 4: Expectation and Belief Propagation Yee Whye Teh ywteh@gatsby.ucl.ac.uk Gatsby Computational Neuroscience Unit University College London http://www.gatsby.ucl.ac.uk/
More informationGaussian processes for inference in stochastic differential equations
Gaussian processes for inference in stochastic differential equations Manfred Opper, AI group, TU Berlin November 6, 2017 Manfred Opper, AI group, TU Berlin (TU Berlin) inference in SDE November 6, 2017
More informationBayesian Inference Course, WTCN, UCL, March 2013
Bayesian Course, WTCN, UCL, March 2013 Shannon (1948) asked how much information is received when we observe a specific value of the variable x? If an unlikely event occurs then one would expect the information
More informationL 0 methods. H.J. Kappen Donders Institute for Neuroscience Radboud University, Nijmegen, the Netherlands. December 5, 2011.
L methods H.J. Kappen Donders Institute for Neuroscience Radboud University, Nijmegen, the Netherlands December 5, 2 Bert Kappen Outline George McCullochs model The Variational Garrote Bert Kappen L methods
More informationLecture 4: Dynamic models
linear s Lecture 4: s Hedibert Freitas Lopes The University of Chicago Booth School of Business 5807 South Woodlawn Avenue, Chicago, IL 60637 http://faculty.chicagobooth.edu/hedibert.lopes hlopes@chicagobooth.edu
More informationWeek 3: The EM algorithm
Week 3: The EM algorithm Maneesh Sahani maneesh@gatsby.ucl.ac.uk Gatsby Computational Neuroscience Unit University College London Term 1, Autumn 2005 Mixtures of Gaussians Data: Y = {y 1... y N } Latent
More informationThe concentration of a drug in blood. Exponential decay. Different realizations. Exponential decay with noise. dc(t) dt.
The concentration of a drug in blood Exponential decay C12 concentration 2 4 6 8 1 C12 concentration 2 4 6 8 1 dc(t) dt = µc(t) C(t) = C()e µt 2 4 6 8 1 12 time in minutes 2 4 6 8 1 12 time in minutes
More informationarxiv: v3 [math.oc] 18 Jan 2012
Optimal control as a graphical model inference problem Hilbert J. Kappen Vicenç Gómez Manfred Opper arxiv:0901.0633v3 [math.oc] 18 Jan 01 Abstract We reformulate a class of non-linear stochastic optimal
More informationBayesian Machine Learning
Bayesian Machine Learning Andrew Gordon Wilson ORIE 6741 Lecture 2: Bayesian Basics https://people.orie.cornell.edu/andrew/orie6741 Cornell University August 25, 2016 1 / 17 Canonical Machine Learning
More informationAdvanced Computational Methods in Statistics: Lecture 5 Sequential Monte Carlo/Particle Filtering
Advanced Computational Methods in Statistics: Lecture 5 Sequential Monte Carlo/Particle Filtering Axel Gandy Department of Mathematics Imperial College London http://www2.imperial.ac.uk/~agandy London
More informationAn Brief Overview of Particle Filtering
1 An Brief Overview of Particle Filtering Adam M. Johansen a.m.johansen@warwick.ac.uk www2.warwick.ac.uk/fac/sci/statistics/staff/academic/johansen/talks/ May 11th, 2010 Warwick University Centre for Systems
More informationContrastive Divergence
Contrastive Divergence Training Products of Experts by Minimizing CD Hinton, 2002 Helmut Puhr Institute for Theoretical Computer Science TU Graz June 9, 2010 Contents 1 Theory 2 Argument 3 Contrastive
More informationL09. PARTICLE FILTERING. NA568 Mobile Robotics: Methods & Algorithms
L09. PARTICLE FILTERING NA568 Mobile Robotics: Methods & Algorithms Particle Filters Different approach to state estimation Instead of parametric description of state (and uncertainty), use a set of state
More informationFast MCMC sampling for Markov jump processes and extensions
Fast MCMC sampling for Markov jump processes and extensions Vinayak Rao and Yee Whye Teh Rao: Department of Statistical Science, Duke University Teh: Department of Statistics, Oxford University Work done
More informationReal Time Stochastic Control and Decision Making: From theory to algorithms and applications
Real Time Stochastic Control and Decision Making: From theory to algorithms and applications Evangelos A. Theodorou Autonomous Control and Decision Systems Lab Challenges in control Uncertainty Stochastic
More informationPart 2: Multivariate fmri analysis using a sparsifying spatio-temporal prior
Chalmers Machine Learning Summer School Approximate message passing and biomedicine Part 2: Multivariate fmri analysis using a sparsifying spatio-temporal prior Tom Heskes joint work with Marcel van Gerven
More informationControlled Diffusions and Hamilton-Jacobi Bellman Equations
Controlled Diffusions and Hamilton-Jacobi Bellman Equations Emo Todorov Applied Mathematics and Computer Science & Engineering University of Washington Winter 2014 Emo Todorov (UW) AMATH/CSE 579, Winter
More informationPolicy Search for Path Integral Control
Policy Search for Path Integral Control Vicenç Gómez 1,2, Hilbert J Kappen 2,JanPeters 3,4, and Gerhard Neumann 3 1 Universitat Pompeu Fabra, Barcelona Department of Information and Communication Technologies,
More informationTowards a Bayesian model for Cyber Security
Towards a Bayesian model for Cyber Security Mark Briers (mbriers@turing.ac.uk) Joint work with Henry Clausen and Prof. Niall Adams (Imperial College London) 27 September 2017 The Alan Turing Institute
More informationAn Implementation of Dynamic Causal Modelling
An Implementation of Dynamic Causal Modelling Christian Himpe 24.01.2011 Overview Contents: 1 Intro 2 Model 3 Parameter Estimation 4 Examples Motivation Motivation: How are brain regions coupled? Motivation
More informationApplications of Optimal Stopping and Stochastic Control
Applications of and Stochastic Control YRM Warwick 15 April, 2011 Applications of and Some problems Some technology Some problems The secretary problem Bayesian sequential hypothesis testing the multi-armed
More informationUnsupervised Learning
Unsupervised Learning Bayesian Model Comparison Zoubin Ghahramani zoubin@gatsby.ucl.ac.uk Gatsby Computational Neuroscience Unit, and MSc in Intelligent Systems, Dept Computer Science University College
More informationStochastic optimal control theory
Stochastic optimal control theory ICML, Helsinki 8 tutorial H.J. Kappen, Radboud University, Nijmegen, the Netherlands July 4, 8 Abstract Control theory is a mathematical description of how to act optimally
More informationBayesian Machine Learning - Lecture 7
Bayesian Machine Learning - Lecture 7 Guido Sanguinetti Institute for Adaptive and Neural Computation School of Informatics University of Edinburgh gsanguin@inf.ed.ac.uk March 4, 2015 Today s lecture 1
More informationImproved diffusion Monte Carlo
Improved diffusion Monte Carlo Jonathan Weare University of Chicago with Martin Hairer (U of Warwick) October 4, 2012 Diffusion Monte Carlo The original motivation for DMC was to compute averages with
More informationIEOR E4570: Machine Learning for OR&FE Spring 2015 c 2015 by Martin Haugh. The EM Algorithm
IEOR E4570: Machine Learning for OR&FE Spring 205 c 205 by Martin Haugh The EM Algorithm The EM algorithm is used for obtaining maximum likelihood estimates of parameters when some of the data is missing.
More informationExercises Tutorial at ICASSP 2016 Learning Nonlinear Dynamical Models Using Particle Filters
Exercises Tutorial at ICASSP 216 Learning Nonlinear Dynamical Models Using Particle Filters Andreas Svensson, Johan Dahlin and Thomas B. Schön March 18, 216 Good luck! 1 [Bootstrap particle filter for
More informationWinter 2019 Math 106 Topics in Applied Mathematics. Lecture 1: Introduction
Winter 2019 Math 106 Topics in Applied Mathematics Data-driven Uncertainty Quantification Yoonsang Lee (yoonsang.lee@dartmouth.edu) Lecture 1: Introduction 19 Winter M106 Class: MWF 12:50-1:55 pm @ 200
More informationLecture 8: Bayesian Estimation of Parameters in State Space Models
in State Space Models March 30, 2016 Contents 1 Bayesian estimation of parameters in state space models 2 Computational methods for parameter estimation 3 Practical parameter estimation in state space
More informationF denotes cumulative density. denotes probability density function; (.)
BAYESIAN ANALYSIS: FOREWORDS Notation. System means the real thing and a model is an assumed mathematical form for the system.. he probability model class M contains the set of the all admissible models
More informationPROBABILITY: LIMIT THEOREMS II, SPRING HOMEWORK PROBLEMS
PROBABILITY: LIMIT THEOREMS II, SPRING 218. HOMEWORK PROBLEMS PROF. YURI BAKHTIN Instructions. You are allowed to work on solutions in groups, but you are required to write up solutions on your own. Please
More informationarxiv: v1 [q-bio.nc] 15 Mar 2018
Effective Connectivity from Single Trial fmri Data by Sampling Biologically Plausible Models H.C. Ruiz Euler, H.J. Kappen September 16, 2018 arxiv:1803.05840v1 [q-bio.nc] 15 Mar 2018 Abstract The estimation
More informationVariational Principal Components
Variational Principal Components Christopher M. Bishop Microsoft Research 7 J. J. Thomson Avenue, Cambridge, CB3 0FB, U.K. cmbishop@microsoft.com http://research.microsoft.com/ cmbishop In Proceedings
More informationExpectation propagation for signal detection in flat-fading channels
Expectation propagation for signal detection in flat-fading channels Yuan Qi MIT Media Lab Cambridge, MA, 02139 USA yuanqi@media.mit.edu Thomas Minka CMU Statistics Department Pittsburgh, PA 15213 USA
More informationCalibration of Stochastic Volatility Models using Particle Markov Chain Monte Carlo Methods
Calibration of Stochastic Volatility Models using Particle Markov Chain Monte Carlo Methods Jonas Hallgren 1 1 Department of Mathematics KTH Royal Institute of Technology Stockholm, Sweden BFS 2012 June
More informationStochastic Generative Hashing
Stochastic Generative Hashing B. Dai 1, R. Guo 2, S. Kumar 2, N. He 3 and L. Song 1 1 Georgia Institute of Technology, 2 Google Research, NYC, 3 University of Illinois at Urbana-Champaign Discussion by
More informationThese slides follow closely the (English) course textbook Pattern Recognition and Machine Learning by Christopher Bishop
Music and Machine Learning (IFT68 Winter 8) Prof. Douglas Eck, Université de Montréal These slides follow closely the (English) course textbook Pattern Recognition and Machine Learning by Christopher Bishop
More informationVariational Inference (11/04/13)
STA561: Probabilistic machine learning Variational Inference (11/04/13) Lecturer: Barbara Engelhardt Scribes: Matt Dickenson, Alireza Samany, Tracy Schifeling 1 Introduction In this lecture we will further
More informationDeterministic Dynamic Programming
Deterministic Dynamic Programming 1 Value Function Consider the following optimal control problem in Mayer s form: V (t 0, x 0 ) = inf u U J(t 1, x(t 1 )) (1) subject to ẋ(t) = f(t, x(t), u(t)), x(t 0
More informationSequential Monte Carlo Methods in High Dimensions
Sequential Monte Carlo Methods in High Dimensions Alexandros Beskos Statistical Science, UCL Oxford, 24th September 2012 Joint work with: Dan Crisan, Ajay Jasra, Nik Kantas, Andrew Stuart Imperial College,
More informationIntroduction to Probabilistic Graphical Models: Exercises
Introduction to Probabilistic Graphical Models: Exercises Cédric Archambeau Xerox Research Centre Europe cedric.archambeau@xrce.xerox.com Pascal Bootcamp Marseille, France, July 2010 Exercise 1: basics
More informationHST 583 FUNCTIONAL MAGNETIC RESONANCE IMAGING DATA ANALYSIS AND ACQUISITION A REVIEW OF STATISTICS FOR FMRI DATA ANALYSIS
HST 583 FUNCTIONAL MAGNETIC RESONANCE IMAGING DATA ANALYSIS AND ACQUISITION A REVIEW OF STATISTICS FOR FMRI DATA ANALYSIS EMERY N. BROWN AND CHRIS LONG NEUROSCIENCE STATISTICS RESEARCH LABORATORY DEPARTMENT
More informationCS Lecture 19. Exponential Families & Expectation Propagation
CS 6347 Lecture 19 Exponential Families & Expectation Propagation Discrete State Spaces We have been focusing on the case of MRFs over discrete state spaces Probability distributions over discrete spaces
More informationContinuous Time Particle Filtering for fmri
Continuous Time Particle Filtering for fmri Lawrence Murray School of Informatics University of Edinburgh lawrence.murray@ed.ac.uk Amos Storkey School of Informatics University of Edinburgh a.storkey@ed.ac.uk
More informationLinearly-Solvable Stochastic Optimal Control Problems
Linearly-Solvable Stochastic Optimal Control Problems Emo Todorov Applied Mathematics and Computer Science & Engineering University of Washington Winter 2014 Emo Todorov (UW) AMATH/CSE 579, Winter 2014
More informationThe Particle Filter. PD Dr. Rudolph Triebel Computer Vision Group. Machine Learning for Computer Vision
The Particle Filter Non-parametric implementation of Bayes filter Represents the belief (posterior) random state samples. by a set of This representation is approximate. Can represent distributions that
More informationNumerical Solutions of ODEs by Gaussian (Kalman) Filtering
Numerical Solutions of ODEs by Gaussian (Kalman) Filtering Hans Kersting joint work with Michael Schober, Philipp Hennig, Tim Sullivan and Han C. Lie SIAM CSE, Atlanta March 1, 2017 Emmy Noether Group
More informationThe Kalman Filter ImPr Talk
The Kalman Filter ImPr Talk Ged Ridgway Centre for Medical Image Computing November, 2006 Outline What is the Kalman Filter? State Space Models Kalman Filter Overview Bayesian Updating of Estimates Kalman
More informationLinear-Quadratic-Gaussian (LQG) Controllers and Kalman Filters
Linear-Quadratic-Gaussian (LQG) Controllers and Kalman Filters Emo Todorov Applied Mathematics and Computer Science & Engineering University of Washington Winter 204 Emo Todorov (UW) AMATH/CSE 579, Winter
More informationSequential Monte Carlo methods for system identification
Technical report arxiv:1503.06058v3 [stat.co] 10 Mar 2016 Sequential Monte Carlo methods for system identification Thomas B. Schön, Fredrik Lindsten, Johan Dahlin, Johan Wågberg, Christian A. Naesseth,
More informationKolmogorov Equations and Markov Processes
Kolmogorov Equations and Markov Processes May 3, 013 1 Transition measures and functions Consider a stochastic process {X(t)} t 0 whose state space is a product of intervals contained in R n. We define
More informationStochastic Optimal Control in Continuous Space-Time Multi-Agent Systems
Stochastic Optimal Control in Continuous Space-Time Multi-Agent Systems Wim Wiegerinck Bart van den Broek Bert Kappen SNN, Radboud University Nijmegen 6525 EZ Nijmegen, The Netherlands {w.wiegerinck,b.vandenbroek,b.kappen}@science.ru.nl
More informationMonte Carlo Approximation of Monte Carlo Filters
Monte Carlo Approximation of Monte Carlo Filters Adam M. Johansen et al. Collaborators Include: Arnaud Doucet, Axel Finke, Anthony Lee, Nick Whiteley 7th January 2014 Context & Outline Filtering in State-Space
More informationDynamic Causal Modelling for fmri
Dynamic Causal Modelling for fmri André Marreiros Friday 22 nd Oct. 2 SPM fmri course Wellcome Trust Centre for Neuroimaging London Overview Brain connectivity: types & definitions Anatomical connectivity
More informationUse of probability gradients in hybrid MCMC and a new convergence test
Use of probability gradients in hybrid MCMC and a new convergence test Ken Hanson Methods for Advanced Scientific Simulations Group This presentation available under http://www.lanl.gov/home/kmh/ June
More informationThe Hierarchical Particle Filter
and Arnaud Doucet http://go.warwick.ac.uk/amjohansen/talks MCMSki V Lenzerheide 7th January 2016 Context & Outline Filtering in State-Space Models: SIR Particle Filters [GSS93] Block-Sampling Particle
More informationThe Origin of Deep Learning. Lili Mou Jan, 2015
The Origin of Deep Learning Lili Mou Jan, 2015 Acknowledgment Most of the materials come from G. E. Hinton s online course. Outline Introduction Preliminary Boltzmann Machines and RBMs Deep Belief Nets
More informationSequential Monte Carlo and Particle Filtering. Frank Wood Gatsby, November 2007
Sequential Monte Carlo and Particle Filtering Frank Wood Gatsby, November 2007 Importance Sampling Recall: Let s say that we want to compute some expectation (integral) E p [f] = p(x)f(x)dx and we remember
More informationCOMP2610/COMP Information Theory
COMP2610/COMP6261 - Information Theory Lecture 9: Probabilistic Inequalities Mark Reid and Aditya Menon Research School of Computer Science The Australian National University August 19th, 2014 Mark Reid
More informationState Space Gaussian Processes with Non-Gaussian Likelihoods
State Space Gaussian Processes with Non-Gaussian Likelihoods Hannes Nickisch 1 Arno Solin 2 Alexander Grigorievskiy 2,3 1 Philips Research, 2 Aalto University, 3 Silo.AI ICML2018 July 13, 2018 Outline
More informationVariational Inference. Sargur Srihari
Variational Inference Sargur srihari@cedar.buffalo.edu 1 Plan of Discussion Functionals Calculus of Variations Maximizing a Functional Finding Approximation to a Posterior Minimizing K-L divergence Factorized
More informationLecture 13 : Variational Inference: Mean Field Approximation
10-708: Probabilistic Graphical Models 10-708, Spring 2017 Lecture 13 : Variational Inference: Mean Field Approximation Lecturer: Willie Neiswanger Scribes: Xupeng Tong, Minxing Liu 1 Problem Setup 1.1
More informationVariational Inference via Stochastic Backpropagation
Variational Inference via Stochastic Backpropagation Kai Fan February 27, 2016 Preliminaries Stochastic Backpropagation Variational Auto-Encoding Related Work Summary Outline Preliminaries Stochastic Backpropagation
More information1. Stochastic Processes and filtrations
1. Stochastic Processes and 1. Stoch. pr., A stochastic process (X t ) t T is a collection of random variables on (Ω, F) with values in a measurable space (S, S), i.e., for all t, In our case X t : Ω S
More informationIntroduction to Machine Learning
Introduction to Machine Learning Brown University CSCI 1950-F, Spring 2012 Prof. Erik Sudderth Lecture 25: Markov Chain Monte Carlo (MCMC) Course Review and Advanced Topics Many figures courtesy Kevin
More informationIntroduction. Stochastic Processes. Will Penny. Stochastic Differential Equations. Stochastic Chain Rule. Expectations.
19th May 2011 Chain Introduction We will Show the relation between stochastic differential equations, Gaussian processes and methods This gives us a formal way of deriving equations for the activity of
More informationGraphical Models and Kernel Methods
Graphical Models and Kernel Methods Jerry Zhu Department of Computer Sciences University of Wisconsin Madison, USA MLSS June 17, 2014 1 / 123 Outline Graphical Models Probabilistic Inference Directed vs.
More informationLecture 2: From Linear Regression to Kalman Filter and Beyond
Lecture 2: From Linear Regression to Kalman Filter and Beyond January 18, 2017 Contents 1 Batch and Recursive Estimation 2 Towards Bayesian Filtering 3 Kalman Filter and Bayesian Filtering and Smoothing
More informationRobust regression and non-linear kernel methods for characterization of neuronal response functions from limited data
Robust regression and non-linear kernel methods for characterization of neuronal response functions from limited data Maneesh Sahani Gatsby Computational Neuroscience Unit University College, London Jennifer
More informationGaussian Process Approximations of Stochastic Differential Equations
Gaussian Process Approximations of Stochastic Differential Equations Cédric Archambeau Centre for Computational Statistics and Machine Learning University College London c.archambeau@cs.ucl.ac.uk CSML
More informationCSC2535: Computation in Neural Networks Lecture 7: Variational Bayesian Learning & Model Selection
CSC2535: Computation in Neural Networks Lecture 7: Variational Bayesian Learning & Model Selection (non-examinable material) Matthew J. Beal February 27, 2004 www.variational-bayes.org Bayesian Model Selection
More information