arxiv: v1 [cond-mat.stat-mech] 1 Dec 2017

Transcription

1 Suscetibility Proagation by Using Diagonal Consistency Muneki Yasuda Graduate School of Science Engineering, Yamagata University Kazuyuki Tanaka Graduate School of Information Sciences, Tohoku University arxiv: v1 [cond-mat.stat-mech] 1 Dec 2017 A suscetibility roagation that is constructed by combining a belief roagation a linear resonse method is used for aroximate comutation for Markov rom fields. Herein, we formulate a new, imroved suscetibility roagation by using the concet of a diagonal matching method that is based on mean-field aroaches to inverse Ising roblems. The roosed suscetibility roagation is robust for various network structures, it is reduced to the ordinary suscetibility roagation to the adative Thouless-Anderson-Palmer equation in secial cases. Keywords: belief roagation, linear resonse relation, suscetibility roagation, adative Thouless- Anderson-Palmer equation, diagonal trick method I. INTRODUCTION There is an increased dem for techniques that can be used to evaluate local statistical quantities such as local magnetizations local suscetibilitiescovariances) of sin systems with finite sizes in statistical hysics as well as in other scientific fields. This is because the use alication of Markov rom fields robabilistic oerations is widely revalent in many areas, articularly in the field of information sciences [1, 2]. Linear resonse methods have been used to aroximately evaluate air correlations between non-nearest airs by combining an advanced mean-field method, articularly the cluster variation method, with the linear resonse theory [3, 4]. In the current decade, a suitable technique called suscetibility roagation ) also called a variational linear resonse in the field of information sciences), has been develoed to comute the local suscetibilities of Markov rom fields [5 8]. In general, s are constructed by combining belief roagation algorithms linear resonse methods. Belief roagation algorithms are one of the most oular message-assing tye of algorithms that is used to aroximately comute local magnetizations of Markov rom fields [9], they are equivalent to Bethe aroximations [] used in statistical hysics [11, 12]. It is known that s can be used to comute local suscetibilities with a high degree of accuracy because linear resonse methods artially restore the effects of loos of networks that are neglected in the Bethe aroximations[13]. s have the following inconsistency due to aroximation. In Ising models, the variances, S 2 i, are trivially one because S i {+1, 1}. However, the variances obtained by s are generally not equal to one [13]. We refer this inconsistency as the diagonal inconsistency in this aer. Herein, we roose a new that overcomes the diagonal inconsistency roblem by using the diagonal trick method. The diagonal trick method is a heuristic technique that originatesin the field ofthe inverseisingroblem [14 16]. The roosed aroach is reduced to the conventional on tree system to the adative Thouless-Anderson-Palmer TAP) equation [17, 18] at high temeratures. II. SCHEME OF PROPOSED METHOD A. Gibbs Free Energy Suscetibility Proagation Let us suose that an undirected grah GV,E) with n nodes, where V = {1,2,...,n} is the set of all labels of nodes that E = {i,j)} is the set of all undirected links in the given grah. Let us consider an Ising system S = {S i { 1,+1} i = 1,2,...,n}, whose Hamiltonian is described as HS h,j) := h i S i J ij S i S j 1) i,j) E on the grah GV, E), where the second summation is the summation taken over all the nearest airs. The arameters{j ij }, which exressthe weightsofinteractions, are symmetrical with resect to i j. The Gibbs free energy GFE) of this system, obtained by constraints for local magnetizations, is exressed as [16] { n Gm) := h i m i +max φ i m i φ β 1 ln )} ex βhs φ,j), 2) S where the term β denotes the inverse temerature. It is known that the minimum oint of this GFE with resect to m rovides the Helmholtz free energy of this system, F := β 1 ln S ex βhs h,j) ), that the values of m = {m i i = 1,2,...,n} coincide with the exact local magnetizations of this system at the minimum oint, i.e., F = min m Gm) S = argmin m Gm), where the notation describes the true) exectation in the Ising system 1). Suscetibilities of the

2 2 Ising system, χ ij := S i S j S i S j, are obtained as βχ ij = m i / h j, where m := argmin m Gm). is an aroximation technique that can be used to evaluate the suscetibilities of a given system [5 8]. In the, we emloy an aroximate GFE, G a m), for examle, the naive mean-field GFE the Bethe GFE, instead of the true GFE 2). In the technique, the aroximate suscetibilities are obtained by βχ ij βˆχ ij := ˆm i / h j, where ˆm := argmin m G a m). s have been also used to address the inverse Ising roblem [16, 19]. B. Diagonal Inconsistency Problem Proosed Scheme Since S i {+1, 1}, the quantities χ ii +m i )2 = S 2 i are obviously equal to one. However, it has been known that the quantities obtained by a, i.e., ˆχ ii + ˆm 2 i, are generally not equal to one due to aroximation [13]. In articular, such a diagonal inconsistency can become a serious roblem in attemting to address the inverse Ising roblem [16]. In order to avoid this diagonal inconsistency, a heuristic technique called the diagonal trick method has been emloyed in various mean-field algorithms for address the inverse Ising roblem [14 16]. In this aer, we roose a new scheme of using the concet of the diagonal trick method in which an obtained satisfies the diagonal consistency. The roosed scheme is quite simle can be described as follows: first, with controllable arameters Λ, we extend an aroximated GFE, G a m), as G am,λ) := G a m) 1 2 ) Λ i 1 m 2 i. 3) The second term aears due to the use of the diagonal trick method [16]. Next, we minimize the extended aroximate GFE in Eq. 3) with resect to m formulate a in the general manner as βχ ij Λ) := m i Λ)/ h j, 4) where m Λ) := argmin m G a m,λ). The values of m Λ) χ ij Λ) deend on the values of Λ. Finally, we determine the values of Λ so as to satisfy the diagonal consistencies by using χ ii Λ)+m i Λ)2 = 1. 5) This equation is termed the diagonal matching equation in this aer, because it leads to the matching of the diagonals of the suscetibilities lus the square of local magnetizations) obtained by an aroximation with those obtained by an exact evaluation. An alternative interretation of this equation is rovided in Aendix B. III. IMPROVED SUSCEPTIBILITY PROPAGATION WITH BETHE APPROXIMATION Inthecontextof,theBetheGFEisgenerallyused as the aroximate GFE, i.e. we use G a m) [8, 19]. Herein, we exlicitly formulate a obtained from our roosed scheme by emloying the Bethe GFE. A. Belief Proagation with Λ In the Ising system, the Bethe GFE is exressed as [6, 16, 20] where G a m) = + 1 β + 1 β h i m i 1 i ) i,j) E S i,s j=±1 S i=±1 i,j) E J ij ξ ij ρ 1 S i m i )lnρ 1 S i m i ) ρ 2 S i,s j m i,m j,ξ ij ) lnρ 2 S i,s j m i,m j,ξ ij ), 6) ρ 1 S i m i ) := 1+S im i, 2 ρ 2 S i,s j m i,m j,ξ ij ) := 1+S im i +S j m j +S i S j ξ ij, 4 { ξ ij := coth2βj ij ) m 2 i m2 j )tanh2 2βJ ij ) ) 1/2 } 2m i m j tanh2βj ij ). 7) The notation i denotes the set of nodes connecting to node i, i.e., i := {j i,j) E}. In accordance with Eq. 3), we extend the Bethe GFE by adding the diagonal trick term, we minimize the extended GFE, G am,λ), with resect to m. By using a similar maniulation in references[16, 20], for given a set of arameters Λ, we can obtain m Λ) as the solutions to a set of simultaneous equations given by M i j Λ) = tanh 1{ tanhβj ij )tanh βh i βλ i m i Λ)+ )} M k i Λ) 8) k i\{j} m i Λ) = tanh βh i βλ i m i Λ)+ j im ) j i Λ). 9)

3 3 Equation 8) is known as the message-assing rule of belief roagation. The quantity M i j Λ) is interreted as the message, or the effective field, roagated from node i to node j through the link i,j). If Λ = 0, Eqs. 8) 9) are reduced to the usual belief roagation for Ising systems. B. with Λ Diagonal Matching By using Eq. 4) m Λ) obtained from Eqs. 8) 9), we can formulate a for a given arameter set Λ as follows: βχ ij Λ) = 1 m i Λ)2 1+βΛ i 1 m i Λ)2) βδ i,j + η j) k i ), Λ) k i ) whereδ i,j isthe Kroneckerdelta. The quantitiesη k) i j Λ) are defined as η k) i j Λ) := M i jλ)/ h k, they satisfy the relation ) sinh2βj ij ) βδ i,k β 2 Λ i χ i j Λ) = ik Λ)+ l i\{j} ηk) l i Λ) ). 11) cosh2βj ij )+cosh2 βh i βλ i m i Λ)+ l i\{j} M l iλ) η k) Eqs. ) 11) are obtained by differentiating Eq. 9) with resect to h j Eq. 8) with resect to h k, resectively. If Λ = 0, Eqs. ) 11) are reduced to the usual for Ising systems. To determine the values of Λ that satisfy the diagonal consistencies, we substitute Eq. 5) in Eq. ), which leads to the relations Λ i = 1 β m i Λ)2 j i η i) j i Λ). 12) The roosed, referred to as the imroved suscetibility roagation I-) in this aer, can be comuted by simultaneously solving Eqs. 8), 9), ), 11), 12). From the arguments given in the last aragrahin Aendix B, on a tree grah, the roosed I- is equivalent to the ordinary gives an exact solution, because the Bethe GFE coincides with the true GFE on a tree grah. C. Connection to Adative TAP Equation The roosed I- given in the revious section can be viewed as an extension of the adative TAP equation [17, 18]. The Bethe GFE in Eq. 6) canbe exed as [20] βg a m) = β h i m i β J ij m i m j + S i=±1 i,j) E ρ 1 S i m i )lnρ 1 S i m i )+Oβ 2 ). 13) This exression is known as the naive mean-field GFE. By using the naive mean-field GFE instead of the Bethe GFE in our scheme, we can derive the following simultaneous equations: m i Λ) = tanhβ h i + J ij m j Λ) Λ im i ), Λ) j i 14) βχ ij Λ) = 1 m i Λ)2 1+βΛ i 1 m i Λ)2) βδ i,j +β 2 J ik χ kj ), Λ) 15) k i Λ i = 1 1 m i Λ)2 k i J ik χ ki Λ). 16) Eq. 14), which is known as the mean-field equation, is obtained from the extremal condition of Eq. 13) with resect to m. Eqs. 15) 16) are obtained by using Eqs. 4) 5), resectively. Alternative reresentations of Eqs. 15) 16) are ossible. From Eq. A2), we find that the mean-field suscetibility βχ ij Λ) in Eq. 15) is equivalent to the ijth element of the Hessian matrix of the extended meanfield GFE that is obtained by adding the diagonal trick term to Eq. 13) in accordance with to Eq. 3): βχ ij Λ) = D J ) 1 ij, 17) where the term J denotes the matrix whose ij-th element is J ij the term D denotes the diagonal matrix whose

4 4 ii-th element is defined by D) ii := β 1 1 m i Λ)2 ) 1 + Λ i. Eq. 17) can be also directly obtained from Eq. 15). From Eq. 5) 17), we obtain Belief Proagation: m, M βd βj ) 1 ii = 1 m i Λ)2. 18) Eqs. 17) 18) are the alternative reresentations of Eqs. 15) 16), resectively. By using the shift βλ i +1 m i Λ)2 ) 1 βλ i, Eqs. 14) 18) become m i Λ) = tanhβ h i + j ij ij m j Λ) Λ im i Λ) m i + Λ) ) β1 m i Λ)2 ) βdλ βj ) 1 ii 19) = 1 m i Λ)2, 20) resectively, where the term D Λ denotes the diagonal matrixwhoseii-thelementisλ i. Equations19)20) coincide with the adative TAP equation. Therefore, we find that our I- is equivalent to the adative TAP equation when β is extremely small. Based on the arguments resented in this section, we can reinterret the adative TAP equation as a based on the naive mean-field aroximation with the diagonal trick method. In the context of the adative TAP equation, the arameters Λ are regarded as effects of the Onsager reaction term. Hence, we can exect that our diagonal trick term behaves as an Onsager reaction term. D. Discussion In the conventional scheme, for a given system, we first solve a belief roagation Eqs. 8) 9) with Λ = 0), we obtain solutions to the belief roagation. These solutions are the values of the local magnetizations the messages. Subsequently, to obtain the suscetibilities, we solve a Eqs. ) 11) with Λ = 0) by using the solutions of the belief roagation. Therefore, in the conventional scheme shown to the left in Fig. 1, the results of the cannot affect the belief roagation. In contrast, the roosed scheme shown to the right in Fig. 1 includes feedback from the to the belief roagation through Λ. This feedback marks a significant difference between the conventional scheme the roosed scheme. If one emloys a synchronous udating algorithm, the comutational cost of solving the usual is On E ). Since Eq. 12) does not contribute to the order of the comutational cost, when using a synchronous udating algorithm, the comutational cost of solving the I- is the same as that of solving the usual. a) FIG. 1. Illustration of schemes of a) conventional b) I-. The Bethe aroximation, which is equivalent to the belief roagation, is regarded as the aroximation that neglects the effects of all the loos in a given system [21]. It is known that, by alying a to the Bethe aroximation in the conventional manner, the effects of the loos neglected in the Bethe aroximation are artially restored that the restored loo effects imrove the accuracy of the aroximation of suscetibilities [13]. Hence, itcanbeexectedthatthefeedbackinourscheme restores the effects of the loos, revived by a, to a belief roagation, that this imroves not only the but also the belief roagation. Indeed, a similar observation has been made in reference [15] the validity of the observation was examined in the study from a erturbative oint of view. b) IV. NUMERICAL EXPERIMENT In this section, we rovide the results of certain numerical exeriments. To measure the erformances of our aroximations, we use the mean absolute distances MADs) defined by := D V := 1 n := 1 E S i S i a, 21) i,j) E 1 nn 1)/2 E S i S j S i S j a, 22) i,j) E S i S j S i S j a, 23) where the notation a indicates the average value evaluated by an aroximation method. The exression denotes the MAD for correlation functions between nearest-neighbor airs the exression denotes the MAD for correlation functions between non-nearestneighbor airs. In the following exeriments, the arameters h J are indeendently obtained from the distri-

5 5 butions h h i ) J J ij ) the inverse temerature β is set to the value of one. Let us consider an Ising system on a grah GV,E), with n = 22 h h i ) = Nh i ), 24) J J ij ) = 1 )δj ij )+NJ ij σ 2 /n)), 25) where δx) denotes the Dirac delta function the exressionnx σ 2 )reresentsthestardgaussianwith variance σ 2. In this case, this grah is regarded as a rom grah known as the Erdős-Rényi grah) with the connection robability robability of the link i, j) being resented between nodes i j). This system is identified as the Sherrington-Kirkatrick SK) model when = 1. Figs. 2 3 show the lots of MADs D V,, ) against when σ = 0.3 σ = 0.6, resectively. The curves indicated by,, in these lots are the results obtained via the normal belief roagation, the normal, the adative TAP equation, resectively. Itistobenotedthatin theframework of the Bethe aroximation belief roagation), the correlations { S i S j } between nearest-neighbors can be aroximated by Eq. 7): S i S j ξ ij [20]. The curves indicated by I- in these lots are the results obtained by the roosed I-. In the region where is small, the belief roagation shows good erformance because it is an aroriate for sarse grahs. On the other h, the adative TAP equation shows good erformance in the region where is large, because the adative TAP equation is based on the mean-field method for mean-field models of dense grahs such as SK models. Our method shows the best erformance over all value of. As mentioned reviously, our method includes the normal the adative TAP as secial cases. Hence, our method can be a good aroximation for both: sarse dense systems, it exhibits a robust erformance over a wide range of values of. V. CONCLUSION In this aer, we roosed a new scheme by incororating the concet of the diagonal trick method in the conventional scheme. The I- obtained from the roosed scheme includes the conventional the adative TAP equation as secial cases, it exhibits a robust erformance for various tyes of network structures of given systems, as observed from the results of our numerical exeriments resented in Sec. IV. In rincile, with the I-, different tyes of Hamiltonians can be obtained using Eq. 1), e.g., those reresenting Ising systems with higher-order interactions, different tyes of aroximations from the belief roagation, e.g., generalized belief roagations, can be addressed using our roosed scheme. In our scheme, it is necessary that all rom variables assume binary values of ±1. Hence, the resent scheme cannot immediately be alied to systems with multivalued variables such as Q-Ising systems. Therefore, it is imortant to extend the resent scheme to multivalued systems. This toic will be considered in future studies. Acknowledgment This work was artly suorted by Grants-In-Aid No , No , No ) for Scientific Research from the Ministry of Education, Culture, Sorts, Science Technology, Jaan. The authors thank Dr. Cyril Furtlehner for discussions at various stages of this work. Aendix A: Linear Resonse Relation for Aroximate GFEs Let us consider a function exressed in the form G 0 m) = h i m i +g 0 m). It is be to note that this exression includes all the GFEs the true GFE, the aroximate GFEs, the extended aroximate GFEs) resented in this aer. At the minima of the function, from its extremal condition, the relation h i + g 0m) m i k=1 m=m = 0 A1) holds, where m := argmin m G 0 m). By differentiating Eq. A1) with resect to h j, we obtain 2 g 0 m) δ i,j = m k m i m k k=1 m=m h j 2 G 0 m) = m k m i m k. m=m h j This exression indicates that the relation H 1 0 = χ A2) holds, where the term H 0 denotes the Hessian matrix of G 0 m) whose ij-th element is defined by H 0 ) ij := 2 G 0 m)/ m i m j m=m the term χ denotes the matrix whose ij-th element is defined by χ ) ij := m i / h j. Aendix B: Variational Interretation of Diagonal Matching Equation The diagonal matching equation in Eq. 5) is introduced to overcome the diagonal inconsistency roblem.

6 6 D V * FIG. 2. MADs when σ = 0.3. Each oint is obtained by averaging over 1500 trials. D V * FIG. 3. MADs when σ = 0.6. Each oint is obtained by averaging over 000 trials. In this aendix, we rovide an alternative interretation of the equation. First, we define a measure of closeness between two ositive definite symmetric matrices, A R n n B R n n, in terms of the Kullback-Leibler divergence as where DA B) := N n x Σ) := dxn n x A)ln N nx A) N n x B), B1) 1 2π)n detσ 1 ) ex 1 ) 2 xt Σx reresents the n-dimensional stard Gaussian with the covariant matrix Σ 1. From the roerties of the Kullback-Leibler divergence, DA B) > 0 DA B) = 0 iff A = B is ensured. Let us define the Hessian matrices of the true GFE G a m,λ) as Hm) ) := 2 Gm)/ m ij i m j as H m,λ) ) := 2 G ij am,λ)/ m i m j, resectively, further, let us consider the quantity DHm) H m,λ)) for given values of m. Minimizing DHm) H m,λ)) corresonds to minimizing the quasi-) distance [22] between the true Hessian matrix the aroximate Hessian matrix. The minimum condition of DHm) H m,λ)) with resect to Λ is 1 m 2 i = H m,λ) 1) ii, B2) where we use the fact that Hm) 1) ii = 1 m2 i holds for any m. When m = m Λ), uon using Eq. A2), Eq. B2) yields Eq. 5). Therefore, we can reinterret the diagonal matching equation as the condition of minimization of distance between the true Hessian matrix its aroximation in terms of the Kullback-Leibler divergence at m = m Λ). When G a m) = Gm), minimization of DHm) H m,λ)) obviously leads to Λ = 0. This indicates that the diagonal matching equation in Eq. 5) yields the result Λ = 0, i.e., the added diagonal trick term in Eq. 3) automatically vanishes when G a m) = Gm). [1] M. Oer D. SaadEds.), Advanced Mean Field Methods Theory Practice MIT Press, 2001). [2] M. Mézard A. Montanari, Information, Physics Comutation Oxford University Press, 2009). [3] J. M. Sanchez, Physica A 111, ). [4] T. Morita, Progress of Theoretical Physics 85, ). [5] K. Tanaka, IEICE Trans. on Information Systems

7 7 E86-D, ). [6] M. Welling Y. W. Teh, Artificial Intelligence 143, ). [7] M. Welling Y. W. Teh, Neural Comutation 16, ). [8] M. Mézard T. Mora, Journal of Physiology-Paris 3, ). [9] J. Pearl, Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference 2nd ed.) San Francisco, CA: Morgan Kaufmann, 1988). [] H. A. Bethe, Proceedings of the Royal Society of London. Series A: Mathematical Physical Sciences 150, ). [11] Y. Kabashima D. Saad, Eurohys. Lett. 44, ). [12] J. S. Yedidia, W. T. Freeman, Y. Weiss, Neural Information Processing Systems NIPS) 13, ). [13] M. Yasuda K. Tanaka, J. Phys. A: Math. Theor. 40, ). [14] H. J. Kaen F. B. Rodríguez, Neural Comutation, ). [15] T. Tanaka, Phys. Rev. E 58, ). [16] M. Yasuda K. Tanaka, Neural Comutation 21, ). [17] M. Oer O. Winther, Phys. Rev. Lett. 86, ). [18] M. Oer O. Winther, Phys. Rev. E 64, ). [19] E. Marinari V. Van Kerrebroeck, J. Stat. Mech.: Theor. Ex. 20, P ). [20] T. Horiguchi, Physica A 7, ). [21] M. Yasuda K. Tanaka, J. Phys. Soc. Jn. 75, ). [22] This measure is generally not a distance in a recise mathematical sense, because DA B) DB A).