Estimating the Capacity of the 2-D Hard Square Constraint Using Generalized Belief Propagation

Transcription

1 Estimating the Capacity of the 2-D Hard Square Constraint Using Generalized Belief Propagation Navin Kashyap (joint work with Eric Chan, Mahdi Jafari Siavoshani, Sidharth Jaggi and Pascal Vontobel, Chinese University of Hong Kong) Conference on Applied Mathematics University of Hong Kong August 23, / 41

2 1-D RLL Constraints Let 0 d < k be fixed integers (k is allowed to be ). Definition A binary sequence x 1 x 2... x n {0, 1} n satisfies the (one-dimensional) (d, k)-runlength-limited (RLL) constraint if any pair of successive 1s in the sequence is separated by at least d and at most k 0s. 2 / 41

3 1-D RLL Constraints Let 0 d < k be fixed integers (k is allowed to be ). Definition A binary sequence x 1 x 2... x n {0, 1} n satisfies the (one-dimensional) (d, k)-runlength-limited (RLL) constraint if any pair of successive 1s in the sequence is separated by at least d and at most k 0s. Codes consisting of sequences satisfying a (d, k)-rll constraint are used for writing information on magnetic and optical recording devices such as hard drives and CDs/DVDs. The maximum rate of such a code is given by the capacity of the (d, k)-rll constraint, defined as 1 C d,k = lim n n log 2 Z n (d,k) where Z n (d,k) denotes the number of binary length-n sequences satisfying the constraint. 2 / 41

4 The 1-D (1, )-RLL Constraint A binary sequence satisfies the (1, )-RLL constraint if it does not contain 1s in adjacent (i.e., consecutive) positions. Some easy facts about Z n := Z (1, ) n : Z n, n = 1, 2, 3,..., forms a Fibonacci sequence C 1, := lim Z 1 = 2, Z 2 = 3, and Z n = Z n 1 + Z n 2 for all n 3 1 n n log 1+ 2 Z n = log = / 41

5 The 2-D Hard Square Constraint Definition A binary m n array satisfies the 2-D hard square constraint (also called the 2-D (1, )-RLL constraint) if no row or column of the array contains 1s in adjacent positions. Each such array can also be viewed as an independent set in the m n grid graph. 4 / 41

6 The Hard-Square Entropy Constant Let Z m,n denote the number of such m n arrays. It can be shown (for example, using subaddivity arguments) that the limit η = lim m,n 1 mn log 2(Z m,n ) exists. This limit is called the hard-square entropy constant. Open Problem: Determine the hard-square entropy constant η. 5 / 41

7 The Hard-Square Entropy Constant Let Z m,n denote the number of such m n arrays. It can be shown (for example, using subaddivity arguments) that the limit η = lim m,n 1 mn log 2(Z m,n ) exists. This limit is called the hard-square entropy constant. Open Problem: Determine the hard-square entropy constant η. What is known: Various upper and lower bounds, resulting in the numerical estimate [Justesen & Forchhammer, 1999] η = η is computable to within an accuracy of 1 N [Pavlov, 2010] in time polynomial in N 5 / 41

8 Binary Ising Model on the 2-D Lattice m n grid with mn vertices and 2mn (m + n) edges Variable x i {0, 1} at each vertex i Pairwise function f : {0, 1} {0, 1} R + Defines a joint distribution on {0, 1} m n : p(x) = 1 f (x i, x j ) Z i j 6 / 41

9 Binary Ising Model on the 2-D Lattice m n grid with mn vertices and 2mn (m + n) edges Variable x i {0, 1} at each vertex i Pairwise function f : {0, 1} {0, 1} R + Defines a joint distribution on {0, 1} m n : p(x) = 1 f (x i, x j ) Z i j Quantity of interest: Z = x Called the partition function. f (x i, x j ) i j 6 / 41

10 Special Case: Hard-Square Model f (a, b) = 1 (a,b) (1,1) The partition function here is precisely Z m,n. 7 / 41

11 Gibbs Free Energy Define the energy of a configuration x {0, 1} m n to be E(x) = log f (x i, x j ) i j so that p(x) = 1 Z exp( E(x)). 8 / 41

12 Gibbs Free Energy Define the energy of a configuration x {0, 1} m n to be E(x) = log f (x i, x j ) i j so that p(x) = 1 Z exp( E(x)). For an arbitrary probability distribution b(x) on {0, 1} m n, define the average energy U(b) = b(x)e(x) x the entropy H(b) = x b(x) log b(x) the Gibbs free energy F (b) = U(b) H(b) 8 / 41

13 A Variational Principle log Z = min F (b) b Proof: Write F (b) = log Z + x b(x) log b(x) p(x) = log Z + D(b p) 9 / 41

14 The Bethe Free Energy Let (b i,j (x i, x j )) and (b i (x i )) be beliefs defined for all edges i j and vertices i, respectively. These must satisfy the following: the b i,j s and b i s are probability mass functions on {0, 1} 2 and {0, 1}, respectively x j b i,j (x i, x j ) = b i (x i ) (consistency) 10 / 41

15 The Bethe Free Energy Let (b i,j (x i, x j )) and (b i (x i )) be beliefs defined for all edges i j and vertices i, respectively. These must satisfy the following: the b i,j s and b i s are probability mass functions on {0, 1} 2 and {0, 1}, respectively x j b i,j (x i, x j ) = b i (x i ) (consistency) We then define the Bethe average energy U B ({b i,j }, {b i }) = b i,j (a, b) log f (a, b) i j (a,b) {0,1} 2 the Bethe entropy H B ({b i,j }, {b i }) = i j H(b i,j ) i (d i 1)H(b i ) where d i denotes the degree of the vertex i. the Bethe free energy F B ({b i,j }, {b i }) = U B ({b i,j }, {b i }) H B ({b i,j }, {b i }) 10 / 41

16 The Bethe Approximation log Z B := min F B({b i,j }, {b i }) {b i,j },{b i } 11 / 41

17 The Bethe Approximation log Z B := min F B({b i,j }, {b i }) {b i,j },{b i } Theorem (Yedidia, Freeman and Weiss, 2001) Stationary points of the Bethe free energy functional correspond to the beliefs at fixed points of the belief propagation algorithm. 11 / 41

18 Belief Propagation (The Sum-Product Algorithm) m i a m a i i i j a a Message update rules: m i a (x i ) = c N(i)\a m c i(x i ) m a i (x i ) = x j f (x i, x j )m j a (x j ) Beliefs: b i (x i ) a N(i) m a i(x i ) b a (x i, x j ) f (x i, x j )m i a (x i )m j a (x j ) (the use of indicates that the beliefs must be normalized to sum to 1) 12 / 41

19 How Good is the Bethe Approximation? For beliefs {b i,j } and {b i } at a fixed point of the BP algorithm, define Z BP ({b i,j }, {b i }) := exp ( F B ({b i,j }, {b i }) ) Theorem (Wainwright, Jaakkola and Willsky, 2003) At any fixed point of the BP algorithm, Z Z BP ({b i,j }, {b i }) = i j b i,j(x i, x j ) x i b i(x i ) d i 1 13 / 41

20 How Good is the Bethe Approximation? For beliefs {b i,j } and {b i } at a fixed point of the BP algorithm, define Z BP ({b i,j }, {b i }) := exp ( F B ({b i,j }, {b i }) ) Theorem (Wainwright, Jaakkola and Willsky, 2003) At any fixed point of the BP algorithm, Z Z BP ({b i,j }, {b i }) = i j b i,j(x i, x j ) x i b i(x i ) d i 1 Theorem (Chertkov and Chernyak, 2006) At any fixed point of the BP algorithm, Z Z BP ({b i,j }, {b i }) = 1 + a finite series of correction terms 13 / 41

21 How Good is the Bethe Approximation? Theorem (Ruozzi, 2012) For the binary Ising model considered here, Z Z B. 14 / 41

22 Numerical Results for the Hard-Square Model Recall: η = [Plot above is from Sabato and Molkaraie, 2012] 15 / 41

23 Regions and Region Graphs , 1, 3, 4 1, 4 1, 2, 4, 5 2, 5 2, 14, 5, , 4 4 4, 5 5 5, 6 3, 4, 7, 8 4, 8 4, 5, 8, 9 5, 9 5, 6, 9, , 8 8 8, 9 9 9, , 8, 15, 11 8, 11 8, 9, 11, 12 9, 9, 10, 12 12, 16 R = R 0 R 1 R 2, where R 0 : all the 2x2 subgrids R 1 : all the non-boundary edges (intersections of regions in R 0 ) R 2 : all the non-boundary vertices (intersections of regions in R 1 ) 16 / 41

24 Region-Based Beliefs Beliefs b R (x R ) are defined for all regions R R. These must satisfy the following: each b R is a probability mass function on {0, 1} R, where R denotes the size (number of vertices) of the region R; for regions P R, we have x R\P b R (x R ) = b P (x P ) (consistency) 17 / 41

25 Region-Based Free Energy Given a set of beliefs {b R : R R}, we define for each region R R: the average energy of R U R = b R (x R ) log f (x i, x j ) x R {0,1} R i,j R:i j the entropy of b R H R = b R (x R ) log b R (x R ) x R the free energy of R F R = U R H R 18 / 41

26 Region-Based Free Energy Given a set of beliefs {b R : R R}, we define for each region R R: the average energy of R U R = b R (x R ) log f (x i, x j ) x R {0,1} R i,j R:i j the entropy of b R H R = b R (x R ) log b R (x R ) x R the free energy of R F R = U R H R The region-based free energy of the model (R, {b R }) is defined as F R ({b R }) = F R F R + F R R R 0 R R 1 R R 2 18 / 41

27 The Kikuchi Approximation log Z R = min {b R } F R({b R }) 19 / 41

28 The Kikuchi Approximation log Z R = min {b R } F R({b R }) Theorem (Yedidia, Freeman and Weiss, 2001) Stationary points of the region-based free energy functional correspond to the beliefs at fixed points of a generalized belief propagation (GBP) algorithm. 19 / 41

29 The Kikuchi Approximation log Z R = min {b R } F R({b R }) Theorem (Yedidia, Freeman and Weiss, 2001) Stationary points of the region-based free energy functional correspond to the beliefs at fixed points of a generalized belief propagation (GBP) algorithm. 19 / 41

30 GBP Message Updates (The Parent-to-Child Algorithm) 13, 1, 3, 4 1, 4 1, 2, 4, 5 2, 5 2, 14, 5, 6 3, 4 4 4, 5 5 5, 6 3, 4, 4, 7, 8 8 4, 5, 8, 9 5, 9 5, 6, 9, 10 7, 8 8 8, 9 9 9, 10 7, 8, 8, 8, 9, 15, , 12 9, 12 9, 10, 12, 16 m (4,5,8,9) (5,9) (x 5, x 9 ) = x 4,x 8 f (x 4, x 5 )f (x 4, x 8 )f (x 8, x 9 ) (red messages) (green messages) 20 / 41

31 GBP Message Updates (The Parent-to-Child Algorithm) 13, 1, 3, 4 1, 4 1, 2, 4, 5 2, 5 2, 14, 5, 6 3, 4 4 4, 5 5 5, 6 3, 4, 4, 7, 8 8 4, 5, 8, 9 5, 9 5, 6, 9, 10 7, 8 8 8, 9 9 9, 10 7, 8, 8, 8, 9, 15, , 12 9, 12 9, 10, 12, 16 m (4,5) (5) (x 5 ) = x 4 f (x 4, x 5 ) (red messages) 21 / 41

32 GBP Beliefs (The Parent-to-Child Algorithm) 13, 1, 3, 4 1, 4 1, 2, 4, 5 2, 5 2, 14, 5, 6 3, 4 4 4, 5 5 5, 6 3, 4, 4, 7, 8 8 4, 5, 8, 9 5, 9 5, 6, 9, 10 7, 8 8 8, 9 9 9, 10 7, 8, 8, 8, 9, 15, , 12 9, 12 9, 10, 12, 16 b (4,5,8,9) (x 4, x 5, x 8, x 9 ) f (x 4, x 5 )f (x 4, x 8 )f (x 5, x 9 )f (x 8, x 9 ) (red messages) 22 / 41

33 GBP Beliefs (The Parent-to-Child Algorithm) 13, 1, 3, 4 1, 4 1, 2, 4, 5 2, 5 2, 14, 5, 6 3, 4 4 4, 5 5 5, 6 3, 4, 4, 7, 8 8 4, 5, 8, 9 5, 9 5, 6, 9, 10 7, 8 8 8, 9 9 9, 10 7, 8, 8, 8, 9, 15, , 12 9, 12 9, 10, 12, 16 b (4,5) (x 4, x 5 ) f (x 4, x 5 ) (red messages) 23 / 41

34 GBP Beliefs (The Parent-to-Child Algorithm) 13, 1, 3, 4 1, 4 1, 2, 4, 5 2, 5 2, 14, 5, 6 3, 4 4 4, 5 5 5, 6 3, 4, 4, 7, 8 8 4, 5, 8, 9 5, 9 5, 6, 9, 10 7, 8 8 8, 9 9 9, 10 7, 8, 8, 8, 9, 15, , 12 9, 12 9, 10, 12, 16 b (5) (x 5 ) (red messages) 24 / 41

35 How Good is the Kikuchi Approximation? Looking at the hard-square model again... Recall: η = [Plot above is from Sabato and Molkaraie, 2012] 25 / 41

36 What We Conjecture Conjecture (Chan et al., ISIT 14) For the binary Ising model considered here, 1 mn log Z 1 mn log Z R = o(1) where o(1) is a positive term that goes to 0 as m, n. In other words, we conjecture that Z Z R = exp ( mn o(1) ). 26 / 41

37 Attacking the Conjecture: The Opening Gambit For beliefs {b R } at a fixed point of the GBP algorithm, define Z GBP ({b R }) := exp ( F R ({b R }) ) Theorem (Chan et al., ISIT 14) At a fixed point of the GBP algorithm, Z Z GBP ({b R }) = R R 0 R 2 b R (x R ) x R R 1 b R (x R ) 27 / 41

38 Attacking the Conjecture: The Opening Gambit For beliefs {b R } at a fixed point of the GBP algorithm, define Z GBP ({b R }) := exp ( F R ({b R }) ) Theorem (Chan et al., ISIT 14) At a fixed point of the GBP algorithm, Z Z GBP ({b R }) = R R 0 R 2 b R (x R ) x R R 1 b R (x R ) Compare this with Theorem (Wainwright, Jaakkola and Willsky, 2003) At any fixed point of the BP algorithm, Z Z BP ({b i,j }, {b i }) = i j b i,j(x i, x j ) x i b i(x i ) d i 1 27 / 41

39 Question to be Addressed Question For the binary Ising model considered here, is it true that the beliefs {b R } at a fixed point of GBP satisfy 1 mn log R R 0 R 2 b R (x R ) = o(1) x R R 1 b R (x R ) where o(1) is a positive term that goes to 0 as m, n? 28 / 41

40 What We Can Prove... Theorem (Chan et al., ISIT 14) For a binary Ising model of size at most 5 5, or size equal to 3 n (or n 3) we have R R 0 R 2 b R (x R ) 1 R R 1 b R (x R ) at any fixed point of GBP. Consequently, at any fixed point of GBP. x Z Z GBP ({b R }) 29 / 41

41 A Key Tool: Log-Supermodularity A function g : {0, 1} k R + is called log-supermodular if for all x, y {0, 1} k. g(x)g(y) g(x y)g(x y) 30 / 41

42 A Key Tool: Log-Supermodularity A function g : {0, 1} k R + is called log-supermodular if for all x, y {0, 1} k. g(x)g(y) g(x y)g(x y) Log-supermodularity is preserved under multiplication: g 1, g 2 log-supermodular = g 1 g 2 log-supermodular marginalization: g log-supermodular = x 1 g(x) log-supermod (this follows from the Ahlswede-Daykin four functions theorem) 30 / 41

43 Log-Supermodularity of Functions with Binary Inputs A function f : {0, 1} 2 R + is log-supermodular iff f (01)f (10) f (00)f (11) If f : {0, 1} 2 R + is not log-supermodular, then the function is log-supermodular. f (a, b) = f (a, 1 b) 31 / 41

44 A Local Transformation A binary Ising model defined by local functions f is equivalent to a binary Ising model defined by f : The partition functions are equal: Z(f ) = Z( f ) For each set of beliefs {b R }, there exists a corresponding { b R } such that R R 0 R 2 b R (x R ) = R R br 0 R 2 (x R ) x R R 1 b R (x R ) x R R br 1 (x R ) 32 / 41

45 Ising Models with Log-Supermodular Local Functions Lemma In a binary Ising model with log-supermodular local functions, the BP and GBP message update rules preserve log-supermodularity of messages. Thus, if BP and GBP are initialized with log-supermodular messages, then the messages in subsequent iterations of BP and GBP remain log-supermodular, and the BP-based beliefs b i,j, b i and the GBP-based beliefs b R are all log-supermodular. 33 / 41

46 The 2 2 Grid The Kikuchi approximation is exact: Z R = Z. 34 / 41

47 The 2 2 Grid The Kikuchi approximation is exact: Z R = Z. Theorem For the 2 2 grid, at any fixed point of BP, we have Z Z BP ({b i,j }, {b i }) = 1 + 1,2 2,3 3,4 4,1 i b, i(0)b i (1) where i,j = b i,j (00)b i,j (11) b i,j (01)b i,j (10). 34 / 41

48 Proof of 2 2 Theorem Start with Wainwright-Jaakkola-Willsky: Z Z BP ({b i,j }, {b i }) = x 1,...,x 4 = x 1,...,x 4 i j b i,j(x i, x j ) i b i(x i ) b i (x i ) i i j b i,j (x i, x j ) b i (x i )b j (x j ) Verify that where s(0) = 1 and s(1) = +1. Hence, Z Z BP ({b i,j }, {b i }) = b i,j (x i, x j ) b i (x i )b j (x j ) = 1 + s(x i)s(x j ) ij b i (x i )b j (x j ), x 1,...,x 4 b i (x i ) i i j ( 1 + s(x ) i)s(x j ) ij b i (x i )b j (x j ) 35 / 41

49 Proof (cont d) Expand out the product i j Note that and Z Z BP = x 1,...,x 4 All other terms x 1,...,x 4 x 1,...,x 4 b i (x i ) i x 1,...,x 4 ( b i (x i ) i b i (x i ) = i i i j i,j 1 + s(x i )s(x j ) ij b i (x i )b j (x j ) ( i b i(x i ) 2 = i j = i j = i j ) : i j ) i,j i b i(x i ) 2 x i b i (x i ) = 1. 1 i,j x 1,...,x 4 i,j i i,j b i (x i )( ) vanish. i i i b i(x i ) x i 1 b i (x i ) 1 b i (0)b i (1) 36 / 41

50 The 3 3 Grid Theorem For the 3 3 grid, at any fixed point of GBP, the ratio Z/Z GBP ({b R }) is given by ( (0) 1 + b 5 (0) b 51 (00)b 51 (01) ) 4 + b 5 (1)( ) 4 (1), b 51 (10)b 51 (11) where (x) = b 512 (x00)b 512 (x11) b 512 (x01)b 512 (x10). 37 / 41

51 Proof of 3 3 Theorem Start with Z Z GBP ({b R }) = Marginalize out x 6, x 7, x 8, x 9 : Z Z GBP ({b R }) = b(x 1526)b(x 1537)b(x 2548)b(x 3549)b(x 5 ) b(x x 15 )b(x 25 )b(x 35 )b(x 45 ) 1,...,x 9 b(x 152)b(x 153)b(x 254)b(x 354)b(x 5 ) b(x x 15 )b(x 25 )b(x 35 )b(x 45 ) 1,...,x 5 38 / 41

52 Proof (cont d) Now, define and write Z Z GBP({b R }) as B(x) = b(x 15)b(x 25 )b(x 35 )b(x 45 ) b(x 5 ) 3 B(x) b(x 152)b(x 5 ) b(x x 15 )b(x 25 ) b(x 153)b(x 5 ) b(x 15 )b(x 35 ) b(x 254)b(x 5 ) b(x 25 )b(x 45 ) b(x 354)b(x 5 ) b(x 35 )b(x 45 ) 1,...,x 5 Next, verify that b(x i5j )b(x 5 ) b(x i5 )b(x j5 ) = 1 + s(x i)s(x j ) 5ij (x 5 ) b(x i5 )b(x j5 ) where s(0) = 1 and s(1) = +1. Plug this back into the expression for Z Z GBP({b R }) and simplify. 39 / 41

53 Proof (cont d) Upon simplification, we obtain Z Z GBP ({b R }) = 1 + (x 5 ) 4 b(x x 5 ) 3 5 This further simplifies to 4 x 1,...,x 4 i=1 1 b(x i5 ) Z Z GBP ({b R }) = 1 + (x 5 ) 4 ( 1 b(x x 5 ) 3 b(x 5 0) + 1 b(x 5 1) 5 = 1 + ( ) 4 (x 5 ) b(x 5 ) b(x x 5 0)b(x 5 1) 5 ) 4 40 / 41

54 References [1] Chun Lam Chan, Estimating the Partition Function of Binary Pairwise Graphical Models Using Generalized Belief Propagation, M. Phil. Thesis, Dept. Information Engg., Chinese University of Hong Kong, Aug [2] C.L. Chan, M. Jafari Siavoshani, S. Jaggi, N. Kashyap, and P.O. Vontobel, Generalized Belief Propagation for Estimating the Partition Function of the 2D Ising Model, in Proc IEEE Int. Symp. Inf. Theory (ISIT 2015), Hong Kong, China, June / 41