Understanding MCMC. Marcel Lüthi, University of Basel. Slides based on presentation by Sandro Schönborn

Size: px

Start display at page:

Download "Understanding MCMC. Marcel Lüthi, University of Basel. Slides based on presentation by Sandro Schönborn"

Daniel Williamson
5 years ago
Views:

1 Understanding MCMC Marcel Lüthi, University of Basel Slides based on presentation by Sandro Schönborn 1

2 The big picture which satisfies detailed balance condition for p(x) an aperiodic and irreducable If Markov Chain is a- periodic and irreducable it converges to Markov chain induces Metropolis Hastings Algorithm samples from Equilibrium distribution is Distribution p(x)

3 Understanding Markov Chains 3

4 Markov Chain State space Sequence of random variables X N i i=1, X i S with joint distribution P X 1, X 2,, X N = P X 1 P(X i X i 1 ) N i=2 Transition probability Initial distribution 1/6 1/3 2 Simplifications: (for our analysis) Discrete state space: S = {1, 2,, K} Automatically true if we use computers (e.g. 32 bit floats) Homogeneous Chain: P X i = l X i 1 = m = T lm 1/2 3 4

5 Example: Markov Chain Simple weather model: dry (D) or rainy (R) hour Condition in next hour? X t+1 State space S = {D, R} Stochastic: P(X t+1 X t ) Depends only on current condition X t D 0.2 R Draw samples from chain: Initial: X 0 = D Evolution: P X t+1 X t DDDDDDDDRRRRRRRRRRRDDDDDDDDDDD DDDDDDDDDDDDDDDDDDDDDDDDDDDDDD DDDDDDDDDRDD... Long-term Behavior Does it converge? Average probability of rain? Dynamics? How quickly will it converge? 5

6 Discrete Homogeneous Markov Chain Formally linear algebra: Distribution (vector): P X i : p i = P(X i = 1) P(X i = K) Transition probability (transition matrix): P X i X i 1 : T = P 1 1 P 1 K P K 1 P K K T lm = P l m = P X i = l X i 1 = m 6

7 Evolution of the Initial Distribution Evolution of P X 1 P(X 2 ): Evolution of n steps: P X 2 = l = P l m P X 1 = m m S p 2 = Tp 1 p n+1 = T n p 1 Is there a stable distribution p? (steady-state) p = Tp A stable distribution is an eigenvector of T with eigenvalue λ = 1 7

8 Steady-State Distribution: p It exists: T subject to normalization constraint: left eigenvector to eigenvalue 1 T lm = T = 1 1 l T has eigenvalue λ = 1 (left-/right eigenvalues are the same) Steady-state distribution as corresponding right eigenvector Tp = p Does any arbitrary initial distribution evolve to p? Convergence? Uniqueness? 8

9 Equilibrium Distribution: p Additional requirement for T: T n lm > 0 for n > N 0 The chain is called irreducible and aperiodic (implies ergodic) All states are connected using at most N 0 steps Return intervals to a certain state are irregular Perron-Frobenius theorem for positive matrices: PF1: λ 1 = 1 is a simple eigenvalue with 1d eigenspace (uniqueness) PF2: λ 1 = 1 is dominant, all λ i < 1, i 1 (convergence) p is a stable attractor, called equilibrium distribution Tp = p 9

10 Convergence Time evolution of arbitrary distribution p 0 p n = T n p 0 Expand p 0 in Eigen basis of T: Te i = λ i e i, λ i < λ 1 = 1, λ k λ k+1 K p 0 = c i e i i K Tp 0 = c i λ i e i i K T n p 0 = c i λ n i e i = c 1 e 1 + λ n 2 c 2 e 2 + λ n 3 c 3 e 3 + i 10

11 Convergence (II) K T n p 0 = c i λ n i e i = c 1 e 1 + λ n 2 c 2 e 2 + λ n 3 c 3 e 3 + (n 1) We have convergence: Rate of convergence: i p + λ 2 n c 2 e 2 T n p 0 n p p n p λ 2 n c 2 e 2 = λ 2 n c 2 c 1 e 1 = p Normalizations: e 1 = 1 σ i p i = 1 11

12 Example: Weather Dynamics Rain forecast for stable versus mixed weather: stable W s = W D R mixed m = p = Long-term average probability of rain: 20% p = Eigenvalues: 1, 0.75 Eigenvalues: 1, 0.25 Rainy now, next hours? RRRRDDDDDDDDDDDD DDDDDDDDDDDDD... Rainy now, next hours? RDDDDDDDDDDDDDDD RDDDRDDDDDDDD... 12

13 Markov Chain: First Results Aperiodic and irreducible chains are ergodic: (every state reachable after > N steps, irregular return time) Convergence towards a unique equilibrium distribution p Equilibrium distribution p Eigenvector of T with eigenvalue λ = 1: Rate of convergence: Tp = p Exponential decay with second largest eigenvalue λ 2 n Only useful if we can design chain with desired equilibrium distribution! 13

14 Detailed Balance Special property of some Markov chains Distribution p satisfies detailed balance if the total flow of probability between every pair of states is equal, (we have a local equilibrium): P l m p m = P m l p l Detailed balance implies: p is the equilibrium distribution Tp l = T lm p m = T ml p l = p l m m Most MCMC methods construct chains which satisfies detailed balance. 14

15 The Metropolis-Hastings Algorithm MCMC to draw samples from an arbitrary distribution 16

16 Idea of Metropolis Hastings algorithm Design a Markov Chain, which satisfies the detailed balance condition T MH x x P x = T MH x x P x Ergodicity ensures that chain converges to this distribution

17 Attempt 1: A simple algorithm Initialize with sample x Generate next sample, with current sample x 1. Draw a sample x from Q(x x) ( proposal ) 2. Emit current state x as sample It s a Markov chain Need to choose Q for every P to satisfy detailed balance Q x x P x = Q x x P x

18 Attempt 2: More general solution Initialize with sample x Generate next sample, with current sample x 1. Draw a sample x from Q(x x) ( proposal ) 2. With probability α(x, x ) emit x as new sample 3. With probability 1 α(x, x ) emit x as new sample It s a Markov chain Decouples Q from P through acceptance rule a How to choose a?

19 What is the acceptance function a? Case A: x = x T MH x x P x = T MH x x P x a x x Q x x P x = a x x Q x x P x Detailed balance trivially satisfied for every a(x,x) Case B: x x We have the following requirement a x x a x x = Q x x P x Q x x P x

20 What is the acceptance function a? Requirement: Choose a(x x) such that a x x a x x = Q x x P x Q x x P x a x x is probability distribution a x x 1 and a x x 0 Easy to check that: satisfies this property. a x x = min 1, Q x x P x Q x x P x

21 The big picture which satisfies detailed balance condition for p(x) an aperiodic and irreducable If Markov Chain is a- periodic and irreducable it converges to Markov chain induces Metropolis Hastings Algorithm samples from Equilibrium distribution is Distribution p(x)

Convex Optimization CMU-10725

Convex Optimization CMU-10725 Simulated Annealing Barnabás Póczos & Ryan Tibshirani Andrey Markov Markov Chains 2 Markov Chains Markov chain: Homogen Markov chain: 3 Markov Chains Assume that the state