INFORMATION GEOMETRY OF PROPAGATION ALGORITHMS AND APPROXIMATE INFERENCE

Transcription

1 2nd Intenational Symposium on Infomation Geomety and its Applications Decembe 12-16, 2005, Tokyo Pages INFORMATION GEOMETRY OF PROPAGATION ALGORITHMS AND APPROXIMATE INFERENCE SHIRO IKEDA 1 intoduction Conside the infeence poblem of undiected gaphical models[8, 9] When the gaph is tee, the Belief Popagation (BP) algoithm (JPeal[11]) is an efficient algoithm and the eact infeence is computed Howeve, when the gaph is loopy, and the loops ae big, the eact infeence becomes intactable Besides sampling methods, such as MCMC, tactable appoimate infeence gives us one pactical solution, and the loopy BP algoithm is one of the most successful methods Recently, it is pointed out that the idea of loopy BP have been used many fields, fo eample, Bethe appoimation[3] in statistical physics, and the decoding algoithms of Low Density Paity Check (LDPC) codes [4] and tubo codes [2] in eo coection codes Although we obseve the loopy BP woks well in many applications, its theoetical aspects ae not fully undestood Among some theoetical studies of loopy BP, we have studied it fom infomation geometical viewpoint[6, 7] In this abstact, we fist summeize the esults of [6, 7] by defining the poblem and showing the popeties loopy BP based on infomation geomety[1] We futhe show its elation to othe popagation algoithms, including the conve concave computational pocedue (CCCP)[14] and Adaptive TAP appoimation[10] 2 Infomation Geomety of BP 21 Poblem, Family of Distibutions, and Pojection Let = ( 1,, n ) T be andom vaiables We conside the case whee each i is binay ie, i { 1, +1} fo simplicity The joint distibution of is q(), and we define the epectation of as ˆη ˆη = E q [] = q() In this aticle, we focus on an infeence poblem of ˆη, which is equivalent to the infeence of n i=1 q( i) In gaphical models, q() is often defined as the poduct of functions {φ i ( i )} and clique functions {φ ( )} as, q() = 1 n φ i ( i ) [ φ ( ) = ep h + Z q i=1 C L c () ψ q ], φ i ( i ) > 0, φ ( ) > 0, hee C is the set of cliques, and L is the cadinality of C h i = 1 2 ln φ i( i = +1) φ i ( i = 1), c () = ln φ ( ), ψ q = ln Z q 1

2 2 S IKEDA Let us conside the following family of pobability distibutions { S = p(; θ, v) = ep[θ + v c() ψ(θ, v)] θ R n, v R L}, c() = (c 1 (),, c L ()) T, v c() = L v c (), whee its natual paamete is (θ, v), θ = (θ n,, θ n ) T, v = (v 1,, v L ) T, and clealy q() = p(; h, 1) S We define M 0 as a submanifold of S specified by v = 0, { M 0 = p 0 (; θ) = ep[h + θ ψ 0 (θ)] } θ R n The poduct of maginal distibutions of q() is included in M 0, that is, n i=1 q( i) M 0 Net we show the definition of the m pojection[1] to M 0 Definition 1 Let π M0 be the opeato of the m pojection to M 0 as follows π M0 () = agmin D[(); p 0 (; θ)] θ Hee, D[ ; ] is the KL (Kullback-Leible)-divegence defined as D[(); p()] = () ln () p() Since M 0 is e flat fom its definition, the m pojection is unique, and we obseve that the infeence of ˆη is equivalent to compute ˆθ = π M0 q(), since ˆη = E p0(; ˆθ) [] 22 Infomation Geomety of BP Although the eact infeence is simply denoted as ˆθ = π M0 q(), the computation becomes intactable fo lage loopy gaphs In many applications, we face with the same poblem, and the BP algoithm is widely used Fo loopy gaphs, the BP algoithm does not necessaily convege, and even if it does, the esult is not equivalent to the eact infeence The following submanifold M plays an impotant ole to undestand BP { [ M = p (; ζ ) = ep h + c () + ζ ψ (ζ )] ζ R n}, = 1,, L M is an e flat submanifold of S, and its natual paamete is ζ We give the infomation geometical view of BP The well-known definition of BP is found somewhee else [9, 11, 12] Infomation geometical view of BP (1) Set t = 0, ζ t = 0, = 1,, L (2) Incement t by one and compute θ and {ζ } iteatively as follows θ t+1 = [π M0 p (; ζ) t ζ], t ζ t+1 = θ t+1 [π M0 p (; ζ t ) ζt ], = 1,, L (3) Repeat step (2) until convegence Let the conveged point of BP be {ζ } and θ, whee θ = ζ /(L 1) We also define ξ = θ ζ The pobability distibution of q(), its final appoimations p 0 (; θ ) M 0, and p (; ζ ) M ae descibed as follows q() = ep[h + c 1 () + + c () + + c L () ψ q ], p 0 (; θ ) = ep[h + ξ ξ + + ξ L ψ 0 (θ )], p (; ζ ) = ep[h + ξ c () + + ξ L ψ (ζ )] The idea of BP is to appoimate c () by ξ in M, taking the infomation fom M ( ) into account The infomation is integated in θ in M 0

3 INFORM GEOM OF PROP ALGO AND APPROXIMATE INFERENCE 3 The following theoem [6] chaacteizes the equilibium of BP Theoem 1 The equilibium (θ, {ζ }) satisfies 1) m condition: θ = π M0 p (; ζ ) 2) e condition: θ = 1 L 1 L ζ We define two submanifolds M and E of S as follows, { M = p() p() S, p() = } p 0 (; θ ) = η 0 (θ ), E = {p() L = Cp 0 (; θ ) t0 p (; ζ L } ) t t = 1, t R, C =0 : nomalization facto Note that M and E ae an m flat and an e flat submanifold, espectively The geometical implications of the 2 conditions ae as follows: m condition: The m flat submanifold M which includes p (; ζ ), = 1,, L, and p 0 (; θ ) is othogonal to M, = 1,, L and M 0, that is, they ae the m pojections to each othe e condition: The e flat submanifold E includes p 0 (; θ 0), p (; ζ ), = 1,, L, and q() Fo tee gaph, we have the following poposition Poposition 1 When q() is epesented with a tee gaph, q(), p 0 (; θ ), and p (; ζ ), = 1,, L ae included in M and E simultaneously This shows when a gaph is a tee, q() and p 0 (; θ ) ae included in M and the fied point of BP is the eact infeence In the case of a loopy gaph, geneally q() / M and the eact infeence is not a fied point of BP 23 Appoimate Infeence Howeve, we still hope that BP gives a good appoimation The diffeence between the eact infeence and the BP solution is egaded as the discepancy between E and M We have given a peliminay analysis in [5, 6], which showed the pincipal tem of the eo is diectly elated to the e cuvatue of M Theoem 2 Let η(θ, v) = E p(;θ,v) [], then, ˆη = E q() [] is appoimated by the decoding esult η(θ, 0) = E p0(;θ )[] as follows ˆη η(θ, 0) + 1 B B s η(θ, v) 2 (θ,v)=(θ,0) whee B = v=0 v i s g i (θ ) θ=θ θ i, Gθv (θ) = { g i (θ )} = θ v 24 Fee Enegy The following fee enegy is impotant ole to undestand the loopy BP (1) F(θ, ζ 1,, ζ L ) = (L 1)ψ 0 (θ) [ ] ψ (ζ ) + ζ (L 1)θ η 0 (θ) The above function is ewitten in tems of the KL divegence as, F(θ, ζ 1,, ζ L ) = D[p 0 (; θ); q()] D[p 0 (; θ); p (; ζ )] + C, whee C is a constant The following theoem is easily deived

4 4 S IKEDA Theoem 3 The equilibium (θ, ζ ) of BP is a citical point of F(θ, ζ 1,, ζ ) Poof By calculating F = 0, ζ we have the m condition, that is p (; ζ ) = p 0 (; θ), By calculating F θ = 0, we ae led to the e condition (L 1)θ = ζ 25 Local Stability of Fied Points Let I 0 (θ) be the Fishe infomation mati of p 0 (; θ), and I (ζ ) be that of p (; ζ ), = 1,, L Since they belong to the eponential family, we have the following elations: I 0 (θ) = θθ ψ 0 (θ), I (ζ ) = ζζ ψ (ζ ) = 1,, L The local stability of the fied points of BP algoithm, is influenced by the updating ode in the step (2) We show the esults when we update all the ζ, = 1,, L simultaneously[6] Theoem 4 Let us define T as follows O I 0 (θ) 1 I 2 (ζ 2 ) E n I 0 (θ) 1 I L (ζ L ) E n T = I 0 (θ) 1 I 1 (ζ 1 ) E n O, I 0 (θ) 1 I 1 (ζ 1 ) E n O whee E n is the n dimensional identity mati When λ i < 1 fo all i, whee λ i ae the eigenvalues of the mati T, the equilibium point is locally stable 3 Relation to Othe Popagation Algoithms 31 CCCP Since the equilibium of BP is chaacteized with the e and the m conditions, thee ae two naive algoithms to find the equilibium One is to constain the paametes always to satisfy the e condition, and seach fo the paametes which satisfy the m condition (e constaint algoithm), the othe is to constain the paametes to satisfy the m condition, and seach fo the paametes which satisfy the e condition (m constaint algoithm) It is easy to see that BP is an e constaint algoithm since the e condition is satisfied at each step, but its convegence is not necessaily guaanteed The othe possibility is the m constaint algoithm We show that CCCP[14] is an m constaint algoithm CCCP is an iteative double loop pocedue to obtain the minimum of an enegy function[13], and the idea was applied to the infeence poblem of loopy gaphs, whee the fee enegy in eq (1) is the enegy function [14] The CCCP algoithm is defined as follows in infomation geometical famewok Infomation geometical view of CCCP inne loop: Given θ t, calculate {ζ t+1 } by solving (2) π M0 p (; ζ t+1 ) = Lθ t ζ t+1, = 1,, L

5 INFORM GEOM OF PROP ALGO AND APPROXIMATE INFERENCE 5 oute loop: Given a set of {ζ t+1 } as the esult of the inne loop, calculate (3) θ t+1 = Lθ t Fom eqs (2) and (3), one obtains ζ t+1 θ t+1 = π M0 p (; ζ t+1 ), = 1,, L, which means that CCCP enfoces the m condition at each iteation On the othe hand, the e condition is satisfied only at the convegent point, which can be easily veified by letting θ t+1 = θ t = θ in eq (3) to yield the e condition (L 1)θ = ζ Theefoe, we can see that the inne and oute loops of CCCP solve the m condition and the e condition, espectively The following theoem [7] shows CCCP has a good convegent popety Theoem 5 Let K be defined as follows K = 1 I0 (θ)i (ζ ) 1 I 0 (θ) L Let λ i i = 1,, n be the eigen values of K, and the equilibium of CCCP is stable when the eigen values of the following K satisfies λ i > 1 1, i = 1,, n L Unde the m constaint, the Hessian of F(θ) at an equilibium point is equal to I0 (θ) [ LK (L 1)E ] I 0 (θ), so that the stability condition fo CCCP is equivalent to the condition that the equilibium is a local minimum of F unde the m constaint 32 Adaptive TAP appoimation We show that the adaptive TAP (Thouless Andeson Palme) appoimation is also fomulated in a simila way by infomation geomety We fist summaize the esults given by Oppe and Winthe[10] Conside q() = 1 n [ ρ( ) ep h + 1 ] (4) Z 2 T J J is a symmetic mati whee diagonal elements ae 0 ρ( ) can take a lot of kinds of functions, and in this etended memo, we conside the case ρ( ) is stictly positive and integable fo R The aim of the adaptive TAP appoach is to infe E q [ ] and E q [ 2 ] Let m be the infeence of E q [ ], and let us define p ( ) as follows, (5) p ( ) = 1 Z () 0 ρ( ) ep [ ( n s=1 J s m s V m + h ) + V 2 2 whee Z () 0 is the nomalization facto to make the integal of p ( ) equal to 1 The summay of adaptive TAP equations is given as follow (6) m = p ( )d [ (S J) 1 ] (7) = ( m ) 2 p ( )d = 2 p ( )d m 2 It is possible to view adaptive TAP equations as an vaiation of BP though infomation geometical famewok of BP Hee, we need to etend i fom binay ],

6 6 S IKEDA vaiable to eal vaiable We conside the case whee ρ( ) is stictly positive fo R Equation (4) can be ewitten as q() = ep[c 0 () + c 1 () + + c n () ψ q ], c 0 () = 1 2 T J, c () = c ( ) = ln ρ( ) + h, ψ q = ln Z Net, we define p 0 as the distibution whose sufficient statistics ae, 2, = 1,, n Natual choice is Gaussian distibution, which is defined as [ p 0 (; µ, S) = ep c 0 () + µ 1 ] 2 T S ψ 0 (µ, S), S = diag(s 1,, s n ), ψ 0 (µ, S) = n 2 ln 2π ln det(s J) µt (S J) 1 µ We set M 0 as M 0 = {p 0 (; µ, S) µ R n, S = diag(s 1,, s n ), s i > 0} Now, ultimate goal of ou poblem is to obtain the m pojection of q() to M 0 Let us define p (; µ \, S \ ) as follows [ p (; µ \, S \ ) = ep c 0 () + c ( ) + µ \ \ 1 ] 2 \ T S \ \ ψ (µ \, S \ ), We define M as S \ = diag(s 1,, s 1, s +1,, s n ) R (n 1) (n 1), µ \ = (µ 1,, µ 1, µ +1,, µ n ) T R n 1, \ = ( 1,, 1, +1,, n ) T R n 1 M = {p (; µ \, S \ )} It is easy to show the e condition holds, q() = 1 n p (; µ \, S \ ) Z p 0 (; µ, S) n 1 Moeove, we can show that the m condition coesponds to the adaptive TAP equations Poposition 2 The m condition is p 0 (; µ, S) = π M0 p (; µ \, S \ ), = 1,, n, and it is satisfied, if and only if the following equations hold (8) m = p ( ; µ \, S \ )d (9) [(S J) 1 ] = 2 p ( ; µ \, S \ )d m 2 = 1,, n, whee m = (S J) 1 µ Now we move to the adaptive TAP equations Lemma 1 Equations (8) and (9) ae equivalent to (6) and (7)

7 INFORM GEOM OF PROP ALGO AND APPROXIMATE INFERENCE 7 Poof Let us set J, K \, and J \ as follows 0 J 1( 1) J 1(+1) J 1n J \ = J ( 1)1 J (+1)1 J n1 J n( 1) J n(+1) 0 K \ = (S \ J \ ), J ( 1)n J (+1)n J = (J 1,, J ( 1), J (+1),, J n ) T We can ewite p 0 (; µ, S) and p (; µ \, S \ ) as follows, (10) p 0 (; µ, S) = p 0 ( ; µ, S)N (K \ 1 (µ \ + J ), K \ 1 ) p (; µ \, S \ ) = p ( ; µ \, S \ )N (K \ 1 (µ \ + J ), K \ 1 ) It is easily poved that p ( ) in (5) is equivalent to p ( ; µ \, S \ ) in (10), which poves the equivalence to the adaptive TAP equations 4 Conclusion We have shown the summay of the infomation geometical famewok to analyze the BP algoithm[6, 7] Since the idea of loopy BP is widely used in many fields, we hope futhe undestanding of BP will be given fom ou famewok We have also shown that othe types of popagation algoithms can be eplained in this famewok This shows the geneality of ou infomation geometical famewok Acknowledgment This wok was suppoted by the Gant-in-Aid fo Young Scientists (B), No , MEXT, Japan Refeences [1] S Amai and H Nagaoka Methods of Infomation Geomety AMS and Ofod Univesity Pess, Povidence, Rhode Island, 2000 [2] C Beou, A Glavieu, and P Thitimajshima Nea Shannon limit eo-coecting coding and decoding: Tubo-codes In Poceedings of IEEE Intenational Confeence on Communications, pages , Geneva, Switzeland, May 1993 [3] H Bethe Statistical theoy of supelattices Poceedings of the Royal Society of London Seies A, Mathematical and Physical Sciences, 150(871): , July 1935 [4] R G Gallage Low density paity check codes IRE Tansactions on Infomation Theoy, IT-8:21 28, Januay 1962 [5] S Ikeda, T Tanaka, and S Amai Infomation geometical famewok fo analyzing belief popagation decode In T G Dietteich, S Becke, and Z Ghahamani, editos, Advances in Neual Infomation Pocessing Systems, volume 14, pages MIT Pess, Cambidge, MA, Apil 2002 [6] S Ikeda, T Tanaka, and S Amai Infomation geomety of tubo and low-density paitycheck codes IEEE Tansactions on Infomation Theoy, 50(6): , June 2004 [7] S Ikeda, T Tanaka, and S Amai Stochastic easoning, fee enegy, and infomation geomety Neual Computation, 16(9): , Septembe 2004 [8] M I Jodan Leaning in Gaphical Models MIT Pess, Cambidge, MA, 1999 [9] S L Lauitzen and D J Spiegelhalte Local computations with pobabilities on gaphical stuctues and thei application to epet systems Jounal of the Royal Statistical Society B, 50: , 1988 [10] M Oppe and O Winthe Adaptive and self-aveaging Thouless-Andeson-Palme mean field theoy fo pobabilistic modeling Physical Review E, 64:056131, 2001 [11] J Peal Pobabilistic Reasoning in Intelligent Systems: Netwoks of Plausible Infeence Mogan Kaufmann, San Mateo CA, 1988 [12] Y Weiss Coectness of local pobability popagation in gaphical models with loops Neual Computation, 12(1):1 41, Januay 2000

8 8 S IKEDA [13] A L Yuille CCCP algoithms to minimize the Bethe and Kikuchi fee enegies: Convegent altenatives to belief popagation Neual Computation, 14(7): , July 2002 [14] A L Yuille and A Rangaajan The concave-conve pocedue Neual Computation, 15(4): , Apil 2003 The Institute of Statistical Mathematics, Minami-Azabu, Minato-ku, Tokyo, , Japan addess: shio@ismacjp URL: