2 Markov Chain Monte Carlo Sampling

Transcription

1 22 Part I. Markov Chais ad Stochastic Samplig Figure 10: Hard-core colourig of a lattice. 2 Markov Chai Mote Carlo Samplig We ow itroduce Markov chai Mote Carlo (MCMC) samplig, which is a extremely importat method for dealig with hard-to-access distributios. Assume that oe eeds to geerate samples accordig to a probability distributio π, but π is too complicated to describe explicitly. A clever solutio is the to costruct a Markov chai that coverges to statioary distributio π, let it ru for a while ad the sample states of the chai. (However, oe obvious problem that this approach raises is determiig how log is for a while? This leads to iterestig cosideratios of the covergece rates ad rapid mixig of Markov chais.) Example 2.1 The hard-core model. A hard-core colourig of a graph G = (V,E) is a mappig such that ξ : V {0,1} (i, j) E ξ(i) = 0 ξ( j) = 0 ( empty vs. occupied sites) (o two occupied sites are adjacet) E.g. o a lattice graph, the hard-core colourig coditio models a exclusio priciple, whereby a particle at oe site excludes the presece of particles at eighbourig sites, cf. Figure 10. I computer sciece terms, a hard-core colourig of a graph G correspods to a idepedet set of odes from G. Deote by µ G the uiform distributio over all the Z G valid hard-core colourigs of G. We would like to sample colourigs accordig to µ G, e.g. i order to compute the expected umber of oes i a valid colourig: E((X)) = (ξ)µ G (ξ) = 1 Z ξ {0,1} V G (ξ)i [ξ is valid], ξ {0,1} V

2 2. Markov Chai Mote Carlo Samplig 23 where (ξ) deotes the umber of oes i colourig ξ. However, the combiatorial structure of distributio µ G is quite complicated; it is far from clear how oe could pick a radom valid hard-core colourig of graph G. (Eve computig their total umber Z G is likely to be a so called #P-complete problem, ad thus ot solvable i polyomial time uless P = NP.) Give a graph G = (V,E), V = {1,...,}, let us cosider the followig Markov chai (X 0,X 1,...) o the space of valid hard-core colourigs of G: Iitially choose X 0 to be ay valid hard-core colourig of G. The, give colourig X t, geerate colourig X t+1 as follows: 1. Choose some ode i V uiformly at radom. 2. If all the eighbours of i have colour 0 i X t, the let X t+1 (i) = 1 with probability 1/2 ad X t+1 (i) = 0 with probability 1/2. 3. At all other odes j, let X t+1 ( j) = X t ( j). It ca be see that the chai thus determied is irreducible (sice all colourigs commuicate via the all-zeros colourig) ad aperiodic (sice for ay colourig ξ, P ξξ > 0). To see that the chai has µ G as its uique statioary distributio, it suffices to check the detailed balace coditios with respect to µ G. Let ξ,ξ be two differet colourigs. If they differ at more tha oe ode, the P ξξ = P ξ ξ = 0, so it suffices to check the case where ξ(i) ξ (i) at a sigle ode i. But the µ G (ξ)p ξξ = 1 Z G = µ G(ξ )P ξ ξ. The above hard-core samplig algorithm is a special case of a Gibbs sampler for a target distributio π o a state space of the form S = C V. The geeral priciple is to choose i step 2 of the state update rule the ew value for X t+1 (i) accordig to the coditioal π-distributio: Pr MC (X t+1 (i) = c) = Pr π (ξ(i) = c ξ( j) = X t ( j), j i). (I additio, the chai eeds to be iitialised i a state X 0 that has ozero π- probability.) It ca be see that the chai so obtaied is aperiodic ad has π as a statioary distributio. Whether the chai is also irreducible depeds o which states ξ have ozero π-probability.

3 24 Part I. Markov Chais ad Stochastic Samplig Example 2.2 Samplig graph k-colourigs. Let G = (V, E) be a graph. The followig is a Gibbs sampler for the uiform distributio i the space S = {1,...,k} V of k-colourigs of G: Iitially choose X 0 to be ay valid k-colourig of G. (Of course, fidig a valid k-colourig is a NP-complete problem for k 3, but let us ot worry about that). The, give colourig X t, geerate colourig X t+1 as follows: 1. Choose some ode i V uiformly at radom. 2. Let C be the set of colours assiged by X t to the eighbours of i: C = {X t ( j) (i, j) E}. (Note that C < k.) Choose a colour for X t+1 (i) uiformly at radom from the set {1,...,k} \C. 3. At all other odes j, let X t+1 ( j) = X t ( j). Note that it is a otrivial questio whether this chai is irreducible or ot. Aother geeral family of MCMC samplers are the Metropolis chais. Let the state space S have some eighbourhood structure, so that it may be viewed as a (large) coected graph (S, N). Deote by N(i) the set of eigbours of state i, ad let d i = N(i). We assume that the eighbourhood structure is symmetric, so that i N( j)if ad oly if j N(i). The the (basic) Metropolis sampler for distributio π o S operates as follows: Iitially choose X 0 to be some state i S. The, give state X t = i, state X t+1 is obtaied as follows: 1. Choose some j N(i) uiformly at radom. { } π 2. With probability mi j d i π i d j,1, accept X t+1 = j. Otherwise let X t+1 = i. Thus, fully writte out the trasitio probabilities are: { } 1 π j d i mi,1, if j N(i) d i π i d j p i j = 0, if j / N(i), j i 1 p i j, if j = i j N(i)

4 2. Markov Chai Mote Carlo Samplig 25 To show that this chai has π as its statioary distributio, it suffices to check the detailed balace coditios: π i p i j = π j p ji i, j S. The coditios are trivial if i = j or j / N(i), so let us cosider the case j N(i). There are two subcases: (i) Case π jd i π i d j 1: The: π i p i j = π i 1 d i 1 π j p ji = π j 1 d j πid j π j d i = π i d i (ii) Case π jd i π i d j < 1: The: π i p i j = π i 1 d i π jd i π i d j = π j d j π j p ji = π j 1 d j 1 (Note that i both cases π i p i j = π j p ji = mi{ π i d i, π j d j }.) Hece π is a statioary distributio of the chai. Furthermore, the chai is guarateed to be aperiodic if there is at least oe i S such that π jd i π i d j < 1 ( p ii > 0) i.e. it is ot the case that for all i, j S: π i d i = π j d j = cost. I the latter case the chai reduces to a simple radom walk o the graph (S,N) with statioary distributio π = [ d1 d d 2 d ] d d as see earlier. Such a radom walk is aperiodic, if ad oly if the graph (S,N) cotais at least oe odd cycle, i.e. if (S,N) is ot bipartite.

5 26 Part I. Markov Chais ad Stochastic Samplig 3 Estimatig the Covergece Rate of a Markov Chai 3.1 Secod Eigevalue, Coductace, Caoical Paths Cosider a regular Markov Chai o state set S = {1,...,}, with trasitio probability matrix P = (p i j ) ad statioary distributio π. We would like to measure the rate of covergece of the chai to π, e.g. i terms of the total variatio distace: where π (i,t) j (i) V (t) = d V (π (i,t),π), = p (t) i j, ad d V (ρ,π) = max ρ(a) π(a) = 1 A S 2 ρ j π j. j S However, we get somewhat tighter results by usig istead of d V the relative poitwise distace drp U ρ j π j (ρ,π) = max. j U π j Hece, we defie our covergece rate fuctio as: U p (t) (t) = max i U du rp (π(i,t) i j π j,π) = max. i, j U π j Whe we cosider covergece over the whole state space, i.e. U = S, we deote simply: (t) = S (t). Propositio 3.1 For ay two distributios ρ, π, where π j > 0 for all j: d V (ρ,π) 1 2 d rp(ρ,π) 1 mi j π j d V (ρ,π). Cosequetly, (i) V (t) 1 2 (t) for all i,t. Defie the mixig time of a give regular chai as τ(ε) = mi{t (t ) ε t t}.

6 3. Estimatig the Covergece Rate of a Markov Chai 27 I algorithmic applicatios, the details of the chai are ofte determied by some iput x, i which case we write x (t), τ x (ε) correspodigly. A chai (more precisely, a family of chais determied by iputs x) is rapidly mixig if ( τ x (ε) = poly x,l 1 ). ε Our goal is ow to establish some techiques for aalysig the covergece rates of Markov chais ad provig them to be rapidly mixig. Lemma 3.2 A regular Markov chai with trasitio matrix P ad statioary distributio π is reversible, if ad oly if the matrix D 1/2 PD 1/2 is symmetric, where D 1/2 = diag( π 1, π 2,..., π ). Proof. D 1/2 PD 1/2 = ( D 1/2 PD 1/2) T DP = P T D. Ispectig this coditio coordiatewise shows that it is exactly the same as the reversibility coditio π i p i j = p ji π j i, j. Now it is easy to see that the matrix A = D 1/2 PD 1/2 has the same eigevalues as P: if λ is a eigevalue of P with left eigevector u, the for the vector v = ud 1/2 we obtai va = ud 1/2 ( D 1/2 PD 1/2) = upd 1/2 = λud 1/2 = λv. Sice matrix A is symmetric for reversible P, this shows that reversible P have real eigevalues. By the Perro-Frobeius theorem they ca thus be ordered as λ 1 = 1 > λ 2 λ 2 λ 3 λ > 1. Deote λ max = ( λ i : 2 i ). Theorem 3.3 Let P be the trasitio matrix of a regular, reversible Markov chai, ad other otatios as above. The for ay U S, U (t) λt max mi π. i i U Proof. Let e 1,...,e be a orthoormal basis for R cosistig of left eigevectors of A, where vector e i is associated to eigevalue λ i. Especially, e 1 = πd 1/2 = [ π 1, π 2,..., π ].

7 28 Part I. Markov Chais ad Stochastic Samplig The A has a spectral represetatio A = i=1 λ i (e i ) T e i = i=1 λ i E i, where E i = (e i ) T e i. Clearly E 2 i = E i, ad E i E j = 0 if i j. Thus, for ay t 0, A t = i=1 λt i E i, ad hece P t = D 1/2 A t D 1/2 = = 1π + λ t i i=2 I compoet form, this meas: p (t) jk = π πk k + π j i=2 i=1 λ t i (D 1/2 (e i ) T )( e i D 1/2) ( D 1/2 (e i ) T )( e i D 1/2). λ t i ei j ei k. Computig the relative poitwise distace covergece rate, we thus get for ay U S: λ t ie i je i k U i=2 (t) = max (5) j,k U π j π k max λ t j,k U e i j ei k i=2 max mi π j j U λt max mi j U π j (by orthoormality of right eigevectors). Theorem 3.4 With otatio ad assumptios as above, (t) λ t max for all eve t. Moreover, if all eigevalues of P are oegative, the the boud holds for all t.

8 3. Estimatig the Covergece Rate of a Markov Chai 29 Proof. Cotiue from equatio (5) above: i=2λ t i(e i j) 2 (t) = S (t) max λ t max max j S j S π j (e i 0 j ) 2 π j, where e i 0 is a eigevector correspodig to eigevalue with absolute value λ max. Necessarily (e i 0 j ) 2 π j for some j for otherwise e i 0 = j=1 (e i 0 j ) 2 < π j = 1, j=1 cotradictig the ormality of e i 0. Note that always λ max = max(λ 2, λ ). Negative eigevalues are ofte a uisace, but they ca always be removed, without affectig the covergece properties of the chai much, by addig appropriate self-loops to the states. E.g.: Propositio 3.5 With otatio ad assumptios as above, cosider the chai determied by trasitio matrix P = 1 2 (I + P). This chai is the also regular ad reversible, has same statioary distributio π, ad its eigevalues satisfy λ > 0 ad λ max = λ 2 = 2 1(1 + λ 2). With Theorem 3.3 ad Propositio 3.5 i mid, it is clear that the key to aalysig covergece rates of reversible Markov chais is to fid good techiques for boudig the secod eigevalue λ 2 away from 1. A iterestig ad ituitive approach to this task is via the otio of coductace of a chai. Give a fiite, regular, reversible Markov chai M o the state space S = {1,...,}, trasitio probability matrix P = (p i j ) ad statioary distributio π = (π i ), we associate to M a weighted graph G = (S,E,W), where E = {(i, j) p i j > 0}, ad the weights o the edges correspod to the ergodic flows betwee states: w i j = π i p i j = π j p ji. Give a state set A S, the capacity of A is defied as C A = π(a) = π i,

9 30 Part I. Markov Chais ad Stochastic Samplig ad the ergodic flow out of A as F A = j / A π i p i j = w i j = w(a,ā). j / A (Note that 0 < F A C A < 1.) The the coductace of the cut (A, Ā), or the (weighted) expasio of A is Φ A = F A C A = w(a,ā) π(a), ad fially the coductace of M, or G, is obtaied as Φ M = Φ(G) = mi Φ A. 0<π(A) 1/2 Sice clearly F A = FĀ for ay A S, this may equally well be defied as: Φ = mi max(φ A,ΦĀ). A S Theorem 3.6 For a regular reversible Markov chai with uderlyig graph G, the secod eigevalue of the trasitio matrix satisfies: (i) λ 2 1 Φ(G)2 2 ; (ii) λ 2 1 2Φ(G). Proof. Later. Corollary 3.7 With otatio ad assumptios as above, the covergece rates for the chai uder cosideratio satisfy, for ay A S ad t 0: (i) ( 1 Φ A 2 /2 ) t (t) mi π ; i

10 3. Estimatig the Covergece Rate of a Markov Chai 31 (ii) (t) (1 2Φ) t. Corollary 3.8 Cosider a family of regular reversible chais where all eigevalues are oegative, parameterised by some iput strig x, ad with uderlyig graphs G x. The the chais are rapidly mixig, if ad oly if Φ(G x ) 1 p( x ), for some polyomial p ad all x. Proof. Accordig to Corollary 3.7 (i): (1 Φ 2 /2) t (t) ε if mi π i ε ) if t l (1 Φ2 l ε + l π mi 2 }{{} Φ 2 /2 if tφ 2 /2 l ε + l π mi if t (l 2 1 Φ 2 ε + l π 1 mi ). Coversely, by Theorem 3.4 ad Corollary 3.7 (ii): (t) > ε if λ t 2 > ε if t lλ 2 > l ε if t l 1 λ < l 1 2 ε ( ) if t 1 λ 2 λ < l 1 2 ε l 1 λ = l λ λ if t < λ 2 1 λ 2 l 1 ε Cosequetly, if t < 1 2Φ 2Φ l 1 ε λ 1 λ, 1 2Φ λ Φ(G x ) l 1 ( 2Φ(G x ) ε τ 2 x(ε) Φ(G x ) 2 l 1 ε + l 1 ) π x. mi 1 λ λ, 0 < λ λ 2

11 32 Part I. Markov Chais ad Stochastic Samplig ½ ½ 1/4 2 1/4 1/4 1 1/4 N eve 1/4 N ½ 1/4 Figure 11: A simple cyclic radom walk. Example 3.1 Simple cyclic radom walk. Cosider the regular, reversible Markov chai described by the graph i Figure 11. Clearly the statioary distributio is π = [ 1, 1,, 1 ]. The coductace Φ A = F A /C A of a cut (A,Ā) is miimised by choosig A to cosist of ay /2 cosecutive odes o the cycle, e.g. A = {1, 2,..., /2}. The Φ = Φ A = F A C A = π i p i j j / A = π i = 1/2 1/2 = 1. Thus, by Theorem 3.6, the secod eigevalue satisfies: 1 2 λ , by Corollary 3.7, the covergece rate satisfies ( 1 ) 2 t ( (t) 1 1 ) t 2 2, ad by Corollary 3.8, the mixig time satisfies: 1 2/ 2/ l 1 τ(ε) 22 (l 1ε ) ε + l ( ) 2 1 l 1 ( ε τ(ε) 22 l + l 1 ). ε Let us ow retur to the proof of Theorem 3.6, establishig the coectio betwee the secod-largest eigevalue ad the coductace of a Markov chai. Recall the statemet of the Theorem: Theorem 3.6 Let M be a fiite, regular, reversible Markov chai ad λ 2 the secod-largest eigevalue of its trasitio probability matrix. The: