Phase transitions in discrete structures

Phase transitions in discrete structures Amin Coja-Oghlan Goethe University Frankfurt

Overview 1 The physics approach. [following Mézard, Montanari 09] Basics. Replica symmetry ( Belief Propagation ). The 1RSB cavity method. 2 Classical rigorous work. The second moment method. Quiet planting. 3 A physics-inspired rigorous approach. The Kauzmann transition. The free entropy in the 1RSB phase. 4 Random k-sat. A rigorous Belief Propagation-based approach. Amin Coja-Oghlan (Frankfurt) Random k-sat 2 / 40

The Boltzmann distribution Let X be a finite set of spins and let N be a large integer. Consider a probability distribution on X N defined by µ(x) = 1 Z M ψ a (x), a=1 with Z = M ψ a (x). x X N a=1 Assume that ψ a depends only on the components a [N] of x. Thus, ψ a (x) = ψ a (x a ). We call µ the Boltzmann distribution and Z the partition function. Amin Coja-Oghlan (Frankfurt) Random k-sat 3 / 40

The factor graph Suppose we are given the Boltzmann distribution µ(x) = 1 Z M ψ a (x a ). a=1 We set up the bipartite factor graph. Square vertices: the factors ψa, a = 1,..., M. Round vertices: the variables x 1,..., x M. Conncect each factor ψ a with all variables x i, i a. Amin Coja-Oghlan (Frankfurt) Random k-sat 4 / 40

The factor graph Example: the 1-dimensional Ising model There are N sites with spins in X = { 1, 1}. For any a {1,..., N 1} we define ψ a (x) = exp(βx a x a+1 ) (β > 0). Hence, a = {a, a + 1}. This yields µ(x) [ M a=1 exp(βx ax a+1 ) = exp β ] N 1 a=1 x ax a+1. Amin Coja-Oghlan (Frankfurt) Random k-sat 5 / 40

The factor graph Example: the Potts antiferromagnet Let X = [K] = {1, 2,..., K}. Let G = (V, E) be a graph on V = [N]. For an edge e joining vertices i, j in G let ψ e (x) = exp [ ] β 1 xi =x j. This gives rise to µ(x) = ψ e (x) = exp [ β #monochromatic edges]. e E Amin Coja-Oghlan (Frankfurt) Random k-sat 6 / 40

The free entropy Generally, the partition function Z scales exponentially with N. Therefore, we are interested in the free entropy 1 N ln Z. In fact, we care mostly about the theormdynamic limit N, i.e., lim N 1 ln Z, N the free entropy density. Does this limit exist? If so, can we compute/approximate it? Is the limit an analytic function, or are there phase transitions? Amin Coja-Oghlan (Frankfurt) Random k-sat 7 / 40

Disordered systems Two levels of randomness The Boltzmann distribution itself is random. Hence, the free entropy density becomes [ ] 1 lim E N N ln Z. Amin Coja-Oghlan (Frankfurt) Random k-sat 8 / 40

Disordered systems Two levels of randomness The Boltzmann distribution itself is random. Hence, the free entropy density becomes [ ] 1 lim E N N ln Z. Example: the Edwards-Anderson model With x as in the 1-dim Ising model, we let [ ] N 1 µ(x) exp β J i x i x i+1. i=1 The J i are independent Gaussian random variables. Amin Coja-Oghlan (Frankfurt) Random k-sat 8 / 40

Disordered systems Example: the diluted mean-filed Potts antiferromagnet Let X = {1,..., K} for some small K 2. Consider a random graph G(N, M) on N vertices with M edges. Amin Coja-Oghlan (Frankfurt) Random k-sat 9 / 40

Disordered systems Example: the diluted mean-filed Potts antiferromagnet Let X = {1,..., K} for some small K 2. Consider a random graph G(N, M) on N vertices with M edges. For any edge e of G(N, M) joining vertices i, j, define ψ e (x) = exp [ ] β 1 xi =x j. The Boltzmann distribution µ(x) e ψ e (x) = exp [ β #monochromatic edges] is random as it depends on the graph G(N, M). Amin Coja-Oghlan (Frankfurt) Random k-sat 9 / 40

Disordered systems Example: coloring random graphs Let X = {1,..., K} for some small K 3. For any edge e of G(N, M) joining vertices i, j, define ψ e (x) = exp [ 1 xi =x j ] = 1xi x j [ zero temperature ]. Then µ(x) e ψ e (x) = 1 no edge is monochromatic. is the uniform distribution over proper K-colorings of G(N, M). Amin Coja-Oghlan (Frankfurt) Random k-sat 10 / 40

Disordered systems Question [Erdős, Rényi 1960] Is there a phase transition in this model (in terms of α = M/N)? Amin Coja-Oghlan (Frankfurt) Random k-sat 10 / 40

Disordered systems Example: random k-sat Let X = {0, 1} and fix some small K 3. Think of x 1,..., x N as Boolean variables. Let Φ be an expression of the form (x 1 x 17 x 29 ) ( x }{{} 11 x 2 x 1 ) }{{} k-clause Φ 1 k-clause Φ 2 with M clauses, chosen uniformly at random. For i = 1,..., M let ψ i (x) = 1 clause Φi evaluates to true. Then µ(x) = M i=1 ψ i(x) is uniform over satisfying assignments. Amin Coja-Oghlan (Frankfurt) Random k-sat 11 / 40

Belief Propagation Goals To compute the free entropy on trees. the Belief Propagation equations. the Bethe free entropy. The replica symmetric ansatz. Belief Propagation as a distributional fixed point equation. application to diluted mean-field models. Amin Coja-Oghlan (Frankfurt) Random k-sat 12 / 40

Example: the 1-dimensional Ising model Let X = {±1}, x X N and µ(x) exp [ β N 1 i=1 x i x i+1 + βb Goal: to compute the marginal µ(x i ) of x i. ] N x i. i=1 Amin Coja-Oghlan (Frankfurt) Random k-sat 13 / 40

Example: the 1-dimensional Ising model For j [N] define two distributions ˆν j, ˆν j on X by ˆν j (x j ) ˆν j (x j ) x 1,...,x j 1 X x j+1,...,x N X [ ] j 1 j 1 exp β x i x i+1 + βb x i, exp β i=1 N 1 i=j x i x i+1 + βb i=1 N i=j+1 x i. Then µ(x j ) ˆν j (x j ) exp(βbx j ) ν j (x j ). Amin Coja-Oghlan (Frankfurt) Random k-sat 14 / 40

Example: the 1-dimensional Ising model We have ˆν j+1 (x j+1 ) = x j X ν j(x j ) exp [βx j x j+1 + βbx j ]. Setting we can rephrase the above as u j = 1 2β ln ˆν j(1) ˆν j ( 1), u j+1 = f (u j + B) with f (x) = β 1 atanh(tanh(β) tanh(βx)). The function f has a unique fixed point u. Hence, for j sufficiently far from the boundary, we find µ(x j ) tanh(β(2u + B)). Amin Coja-Oghlan (Frankfurt) Random k-sat 15 / 40

Belief Propagation on trees Suppose that the factor graph of µ(x) is a tree. Generalising the above example, we define messages recursively by ν (t+1) j a (x j) ˆν (t) b j (x j), b j\{a} ˆν (t) b j (x j) x b\j ψ b (x b ) i b\j ν (t) i b (x i). t counts the number of recursive steps, j is the neighborhood of j, the sum ranges over X b\j, with x j is given upfront. Amin Coja-Oghlan (Frankfurt) Random k-sat 16 / 40

Belief Propagation on trees Theorem Assume that the factor graph is a tree of diameter T. 1 The BP equations have a unique fixed point ν. 2 For t > T we have ν (t) = ν, regardless of the initialisation. 3 For all variables x j we have ν (x j ) = µ(x j ), i.e., the BP marginals coincide with the Boltzmann marginals. Amin Coja-Oghlan (Frankfurt) Random k-sat 17 / 40

Belief Propagation on trees Theorem (the Bethe free entropy) Given the messages ν, define M N F (ν) = F a (ν) + F ia (ν), where a=1 F i (ν) i=1 (i,a) F a (ν) = ln ψ a (x a ) ν i a (x i ), x a i a F i (ν) = ln ˆν b i (x i ), x i b i F ia (ν) = ln x i ν i a (x i )ˆν a i (x i ). If the factor graph is a tree, then ln Z = F (ν ). Amin Coja-Oghlan (Frankfurt) Random k-sat 18 / 40

Belief Propagation on trees On trees, BP converges to its unique fixed point. This fixed point yields the correct marginals, the free entropy. Example: the 1-dim Ising model. Amin Coja-Oghlan (Frankfurt) Random k-sat 19 / 40

Belief Propagation on infinite trees Question Does this formalism survive the thermodynamic limit? That is, does it extend to infinite trees? Amin Coja-Oghlan (Frankfurt) Random k-sat 20 / 40

Belief Propagation on infinite trees Question Does this formalism survive the thermodynamic limit? That is, does it extend to infinite trees? Examples of infinite trees The infinite regular tree. Each variable/factor node has the same number of outgoing edges. The factors ψ a are identical. Infinite random trees. The numbers of outgoing edges is are independent Poisson variables. The factors ψa might have some randomness, too. Generally, we assume that sub-trees are identically distributed. Amin Coja-Oghlan (Frankfurt) Random k-sat 20 / 40

Belief Propagation on infinite trees The distributional BP equations We interpret the Belief Propagation equations ν j a (x j ) ˆν b j (x j ), b j\{a} ˆν b j (x j ) ψ b (x b ) ν i b (x i ). x b\j i b\j as distributional fixed point equations. That is, if ψ b, b, j are i.i.d. from the distribution defining the tree, the ν i b are i.i.d. from a distribution ν over distributions, then ν j a has distribution ν. The Bethe free entropy is determined by ν. Amin Coja-Oghlan (Frankfurt) Random k-sat 21 / 40

Belief Propagation on infinite trees Example: coloring regular trees Consider the graph K-coloring problem on the d-ary tree. By symmetry, all the marginals are identical, i.e., ν j a (x j ) = 1/K for all j, x j. Thus, ν is the measure concentrated on the uniform distribution. Amin Coja-Oghlan (Frankfurt) Random k-sat 22 / 40

Belief Propagation on infinite trees Example: random K-SAT Each variable node has a Po(α) number of children. The factors ψ a are i.i.d. random clauses of length K. The fixed point distribution ν is not known to exist, but is conjectured to be highly non-trivial and not discrete, based on numerical evidence ( population dynamics ). This mirrors the asymmetric combinatorics of the problem. Amin Coja-Oghlan (Frankfurt) Random k-sat 23 / 40

The replica symmetric ansatz Diluted mean-field models Think of the factor graphs in models such as random graph K-coloring, random K-SAT. The factor graph is an Erdős-Rényi-like sparse random graph. Thus, roughly speaking, the vertices are bounded degree, the shortest cycle has length Ω(ln N). In other words, locally the graph looks like the infinite random tree! Amin Coja-Oghlan (Frankfurt) Random k-sat 24 / 40

The replica symmetric ansatz The replica symmetric solution Let ν be the BP fixed point on the tree. Let φ be the resulting Bethe free entropy. Then the replica symmetric prediction is that E[ln Z] φ = lim N N, with Z the partition function in, e.g., random K-SAT. Amin Coja-Oghlan (Frankfurt) Random k-sat 25 / 40

The replica symmetric ansatz Example: ferromagnetic Ising on sparse random graphs [DM 08] The replica symmetric solution is correct. Paramagnetic/ferromagnetic phase transition. Amin Coja-Oghlan (Frankfurt) Random k-sat 26 / 40

The replica symmetric ansatz Example: coloring random graphs Let K = #colors, α = M/N density of the graph. Then Z = #K-colorings of G(N, M). According to the replica symmetric solution, 1 N E[ln Z] 1 ln E[Z] = ln K + α ln(1 1/K). N The col/uncol transition occurs at α = (K 1 2 ) ln K + o K (1). Rigorous work: [ACO, Vilenchik 13] The r.s. solution is correct only for α < (K 1 2 ) ln K ln 2 + o K (1). The col/uncol transition occurs at α (K 1 2 ) ln K 1 2 + o K (1). Amin Coja-Oghlan (Frankfurt) Random k-sat 27 / 40

Multiple Belief Propagation fixed points Random factor graphs In random graph K-coloring, the factor graph is similar to G(N, M). Thus, it s far from being a tree. In effect, there can be multiple BP fixed points. These mirror long range correlations. Amin Coja-Oghlan (Frankfurt) Random k-sat 28 / 40

Multiple Belief Propagation fixed points x i w The reconstruction problem Suppose µ is defined by a factor graph G. Pick a variable x i and let G ω be the depth-ω neighborhood. What is the impact of the boundary on x i? Amin Coja-Oghlan (Frankfurt) Random k-sat 29 / 40

Multiple Belief Propagation fixed points Correlation decay: Gibbs uniqueness Let y ω be a boundary configuration. The worst-case influence of the boundary can be cast as E[sup y ω µ (x i y ω ) µ(x i ) ]. Here the expectation is over the choice of µ. If lim ω we say there is Gibbs uniqueness. lim E[sup µ (x i y ω ) µ(x i ) ] = 0, N y ω Observe that this is a purely local property. Amin Coja-Oghlan (Frankfurt) Random k-sat 30 / 40

Multiple Belief Propagation fixed points Correlation decay: non-reconstruction The average influence of the boundary configuration can be cast as E y ω µ(y ω ) µ (x i y ω ) µ(x i ). If lim ω lim E µ(y ω ) µ (x i y ω ) µ(x i ) = 0, N y ω we say there is non-reconstruction. Otherwise reconstruction is possible. Amin Coja-Oghlan (Frankfurt) Random k-sat 31 / 40

Multiple Belief Propagation fixed points Theorem [Achlioptas, ACO 08] If the density satisfies α = M/N > ( ) 1 2 + ε K ln K, then in random graph K-coloring reconstruction is possible. In fact, suppose we initialise BP with one random K-coloring. Then for Ω(n) variables x i the BP marginals satisfy ν (x i ) = 1. Thus, these variables are locally frozen. Interpretation: µ decomposes into a large number of clusters. Amin Coja-Oghlan (Frankfurt) Random k-sat 32 / 40

The 1RSB cavity method 1-step replica symmetry breaking In the above scenario, µ is a convex combination µ = N w i µ i i=1 of probability measures µ i (corresponding to clusters ). Here w i exp(f i N), with F i the free entropy of µ i. Key idea: perform Belief Propagation on the level of the µ i! The parameter β is replaced by the Parisi parameter y. Amin Coja-Oghlan (Frankfurt) Random k-sat 33 / 40

The 1RSB cavity method Idea A Boltzmann distribution on the µ i. This distribution is defined by P y (µ i ) w y i. In terms of the complexity function Σ(φ) = 1 N ln # {i [N ] : F i φ}, the partition function Ξ(y) of ξ y satisfies Ξ(y) = yφ + Σ(φ), with φ s.t. Σ φ = y. Amin Coja-Oghlan (Frankfurt) Random k-sat 34 / 40

The 1RSB cavity method Belief Propagation one level up What is the factor graph of P y? The old messages ν i a, ˆν a i are the variables. There are factor nodes enforcing the BP equations. Additional factor nodes implement the Parisi parameter y. (This is possible due to Bethe s formula.) On this factor graph, write the Belief Propagation equations. 1RSB cavity equations. Complication: the spins are continuous. The BP fixed point for P y yields Σ(φ). Analogous to the Bethe free entropy. Amin Coja-Oghlan (Frankfurt) Random k-sat 35 / 40

The 1RSB cavity method ical, connd q-col entedstatic in Talarge k [11]. 1RSB (aka condensation, Kauzmann transition). confirming r q-coloring 14], the val- l c l s [14] 10 10 14 15 19 20 α d,+ α d α c α s The geometry of Fig. the 2. Boltzmann distribution Pictorial representation of the different phase transiti known The that replica solutions symmetric of aphase. rcsp. At α d,+ some clusters appear, but for α d,+ putforward Dynamic 1RSB. solutions are split among about e NΣ clusters of size e Ns.I comprise only an exponentially small fraction of solutions. Forα d the set of solutions is dominated by a few large clusters (with str weights), and above α s the problem does not admit solutions an corresponding to two distinct intuitions. According to Amin Coja-Oghlan (Frankfurt) Random k-sat 36 / 40

The 1RSB cavity method Example: for what d is the random d-regular graph K-colorable? Setting y = 0, we consider the distribution P 0 (µ i ) = 1 N, N = #clusters. Amin Coja-Oghlan (Frankfurt) Random k-sat 37 / 40

The 1RSB cavity method Example: for what d is the random d-regular graph K-colorable? Setting y = 0, we consider the distribution P 0 (µ i ) = 1 N, N = #clusters. Suppose that each µ i is characterized by frozen variables. Encode µi as a map ζ i : [N] {1, 2,..., K, }. = unfrozen ( joker color ). This gives rise to SP messages: Qi j (x i ) =P 0 -probability that w/out j, i is frozen to color x i. Qi j (x i ) =P 0 -probability that w/out j, i is unfrozen. Amin Coja-Oghlan (Frankfurt) Random k-sat 37 / 40

The 1RSB cavity method Example: for what d is the random d-regular graph K-colorable? We have the fixed point equations x i\j N (x i ) k i\j Q i j (x i ) = Q k i(x k ) x i\j D k i\j Q k i(x k ) (x i [K]). D contains all (c 1,..., c d 1 ) such that [K] {c 1,..., c d 1 }. N (x) contains all (c 1,..., c d 1 ) D with [K] \ {x} {c 1,..., c d 1 }. Let Q i j ( ) = 1 x i Q i j (x i ). Simplification: it might seem reasonable to assume that Q i j (x i ) = a for some fixed 0 a 1/q and all x i [K]. Qi j ( ) = 1 Ka. The above equation becomes ( q 1 ) r=0 a = ( 1)r K 1 r (1 (r + 1)a) d 1 ( q 1 ). r=0 ( 1)r q r+1 (1 (r + 1)a) d 1 Amin Coja-Oghlan (Frankfurt) Random k-sat 38 / 40

The 1RSB cavity method Example: for what d is the random d-regular graph K-colorable? We obtain the complexity ( 1 N ln #clusters) [ q 1 ( ] q Σ = ln ( 1) )(1 r (r + 1)a) d d r + 1 2 ln(1 Ka2 ) r=0 with a the solution to a = ( q 1 ) r=0 ( 1)r K 1 r (1 (r + 1)a) d 1 ( q 1 ) r=0 ( 1)r q. r+1 (1 (r + 1)a) d 1 Now, K-colorability should occur iff Σ > 0. Asymptotically, this yields a threshold of d K col = (2K 1) ln K 1 + o K (1). Amin Coja-Oghlan (Frankfurt) Random k-sat 39 / 40

Conclusion The cavity method as a non-rigorous formalism. free entropy, phase transitions. There are two variants: the replica symmetric version ( Belief Propagation ). the 1RSB version ( Survey Propagation ). Can we develop a rigorous foundation? Amin Coja-Oghlan (Frankfurt) Random k-sat 40 / 40