Approximating Posterior Distributions in Belief Networks using Mixtures

Transcription

1 Apprxiating Psterir Distributins in Belief Netwrks using Mixtures Christpher M. Bishp Neil Lawrence Neural Cputing Research Grup Dept. Cputer Science & Applied Matheatics Astn University Binningha, B4 7ET, U.K. Ti Jaakkla Michael I. Jrdan Center fr Bilgical and Cputatinal Learning Massachusetts Institute f Technlgy 79 Aherst Street, ElO-243 Cabridge, MA 2139, U.S.A. Abstract Exact inference in densely cnnected Bayesian netwrks is cputatinally intractable, and s there is cnsiderable interest in develping effective apprxiatin schees. One apprach which has been adpted is t bund the lg likelihd using a ean-field apprxiating distributin. While this leads t a tractable algrith, the ean field distributin is assued t be factrial and hence unidal. In this paper we denstrate the feasibility f using a richer class f apprxiating distributins based n ixtures f ean field distributins. We derive an efficient algrith fr updating the ixture paraeters and apply it t the prble f learning in sigid belief netwrks. Our results denstrate a systeatic iprveent ver siple ean field thery as the nuber f ixture cpnents is increased. 1 Intrductin Bayesian belief netwrks can be regarded as a fully prbabilistic interpretatin f feedfrward neural netwrks. Maxiu likelihd learning fr Bayesian netwrks requires the evaluatin f the likelihd functin P(VIO) where V dentes the set f instantiated (visible) variables, and represents the set f paraeters (weights and biases) in the netwrk. Evaluatin f P(VIO) requires suing ver expnentially any cnfiguratins f

2 Apprxiating Psterir Distributins in Belief Netwrks Using Mixtures 417 the hidden variables H, and is cputatinally intractable except fr netwrks with very sparse cnnectivity, such as trees. One apprach is t cnsider a rigrus lwer bund n the lg likelihd, which is chsen t be cputatinally tractable, and t ptiize the del paraeters s as t axiize this bund instead. If we intrduce a distributin Q (H), which we regard as an apprxiatin t the true psterir distributin, then it is easily seen that the lg likelihd is bunded belw by F[Q] = L Q(H) In P(V ~H). (1) {H} Q( ) The difference between the true lg likelihd and the bund given by (1) is equal t the Kullback-Leibler divergence between the true psterir distributin P(HIV) and the apprxiatin Q(H). Thus the crrect lg likelihd is reached when Q(H) exactly equals the true psterir. The ai f this apprach is therefre t chse an apprxiating distributin which leads t cputatinally tractable algriths and yet which is als flexible s as t perit a gd representatin f the true psterir. In practice it is cnvenient t cnsider paraetrized distributins, and then t adapt the paraeters t axiize the bund. This gives the best apprxiating distributin within the particular paraetric faily. 1.1 Mean Field Thery Cnsiderable siplificatin results if the del distributin is chsen t be factrial ver the individual variables, s that Q(H) = ni Q(hd, which gives eanfieid thery. Saul et al. (1996) have applied ean field thery t the prble f learning in sigid belief netwrks (Neal. 1992). These are Bayesian belief netwrks with binary variables in which the prbability f a particular variable Si being n is given by P(S, = Ilpa(S,» =" ( ~ ],;S; + b) (2) where u(z) == (1 + e-z)-l is the lgistic sigid functin, pa(si) dente the parents f Si in the netwrk. and Jij and bi represent the adaptive paraeters (weights and biases) in the del. Here we briefly review the fraewrk f Saul et ai. (1996) since this frs the basis fr the illustratin f ixture delling discussed in Sectin 3. The ean field distributin is chsen t be a prduct f Bernulli distributins f the fr Q(H) = II p,~i (1 _ p,;)l-h i (3) in which we have intrduced ean-field paraeters J.Li. Althugh this leads t cnsiderable siplificatin f the lwer bund, the expectatin ver the lg f the sigid functin. arising fr the use f the cnditinal distributin (2) in the lwer bund (I), reains intractable. This can be reslved by using variatinal ethds (Jaakkla, 1997) t find a lwer bund n F(Q), which is therefre itself a lwer bund n the true lg likelihd. In particular, Saul et al. (1996) ake use f the fllwing inequality (In[l + e Zi ]) ::; ei(zi) + In(e-~iZi + e(1-~;)zi) (4) where Zi is the arguent f the sigid functin in (2), and ( ) dentes the expectatin with respect t the ean field distributin. Again, the quality f the bund can be iprved by adjusting the variatinal paraeter ei. Finally, the derivatives f the lwer bund with respect t the J ij and b i can be evaluated fr use in learning. In suary. the algrith invlves presenting training patterns t the netwrk. and fr each pattern adapting the P,i and ei t give the best apprxiatin t the true psterir within the class f separable distributins f the fr (3). The gradients f the lg likelihd bund with respect t the del paraeters J ij and bi can then be evaluated fr this pattern and used t adapt the paraeters by taking a step in the gradient directin.

3 41 C. M. Bishp, N. LAwrence, T. Jaakkla and M I. Jrdan 2 Mixtures Althugh ean field thery leads t a tractable algrith, the assuptin f a cpletely factrized distributin is a very strng ne. In particular, such representatins can nly effectively del psterir distributins which are uni-dal. Since we expect ulti-dal distributins t be cn, we therefre seek a richer class f apprxiating distributins which nevertheless reain cputatinally tractable. One apprach (Saul and Jrdan, 1996) is t identify a tractable substructure within the del (fr exaple a chain) and then t use ean field techniques t apprxiate the reaining interactins. This can be effective where the additinal interactins are weak r are few in nuber, but will again prve t be restrictive fr re general, densely cnnected netwrks. We therefre cnsider an alternative apprach I based n ixture representatins f the fr M Qix(H) = L aq(hl) (5) =l in which each f the cpnents Q(Hl) is itself given by a ean-field distributin, fr exaple f the fr (3) in the case f sigid belief netwrks. Substituting (5) int the lwer bund (1) we btain F[Qix] = L af[q(hl)] + f(, H) (6) where f(, H) is the utual infratin between the cpnent label and the set f hidden variables H, and is given by Q(Hl) f(,h) = L L aq(hl) In Q. (H)' {H} ix (7) The first tenn in (6) is siply a cnvex cbinatin f standard ean-field bunds and hence is n greater than the largest f these and s gives n useful iprveent ver a single ean-field distributin. It is the secnd ter, i.e. the utual infnnatin, which characterises the gain in using ixtures. Since f(, H) ~, the utual infratin increases the value f the bund and hence iprves the apprxiatin t the true psterir. 2.1 Sthing Distributins As it stands, the utual infnnatin itself invlves a suatin ver the cnfiguratins f hidden variables, and s is cputatinally intractable. In rder t be able t treat it efficiently we first intrduce a set f 'sthing' distributins R(Hl), and rewrite the utual infnnatin (7) in the fr f(, H) - LLaQ(Hl)lnR(Hl) - LaIna {H} - L L aq(hl) In {R(H1) Qix(H) }. () {H} a Q(Hl) It is easily verified that () is equivalent t (7) fr arbitrary R(Hl). We next ake use f the fllwing inequality - In x ~ - ~x + In ~ + 1 (9) lhere we utline the key steps. A re detailed discussin can be fund in Jaakkla and Jrdan (1997).

4 Apprxiating Psterir Distributins in Belief Netwrks Using Mixtures 419 t replace the lgarith in the third ter in () with a linear functin (cnditinally n the cpnent label ). This yields a lwer bund n the utual infratin given by J(,H) ~ J),(,H) where h(,h) I:I:aQ(Hl)lnR(Hl)- I: a In a {H} - I: A I: R(Hl)Qix(H) + I: a InA + 1. (1) {H} With J),(, H) substituted fr J(, H) in (6) we again btain a rigrus lwer bund n the true lg likelihd given by F),[Qix(H)] = I: af[q(hl)] + h(, H). (11) The suatins ver hidden cnfiguratins {H} in (1) can be perfred analytically if we assue that the sthing distributins R(Hl) factrize. In particular, we have t cnsider the fllwing tw suatins ver hidden variable cnfiguratins I: R(Hl)Q(Hlk) II I: R(hil)Q(hilk) ~ 7rR,Q(, k) (12) {H} i h. I: Q(Hl) InR(Hl) I: I: Q(hil) InR(hil) ~f H(QIIRl). (13) {H} h. We nte that the left hand sides f (12) and (13) represent sus ver expnentially any hidden cnfiguratins, while n the right hand sides these have been re-expressed in ters f expressins requiring nly plynial tie t evaluate by aking use f the factrizatin f R(Hl). It shuld be stressed that the intrductin f a factrized fr fr the sthing distributins still yields an iprveent ver standard ean field thery. T see this, we nte that if R(Hl) = cnst. fr all {H, } then J(, H) =, and s ptiizatin ver R(Hl) can nly iprve the bund. 2.2 Optiizing the Mixture Distributin In rder t btain the tightest bund within the class f apprxiating distributins, we can axiize the bund with respect t the cpnent ean-field distributins Q(Hl), the ixing cefficients a, the sthing distributins R(Hl) and the variatinal paraeters A' and we cnsider each f these in turn. We will assue that the chice f a single ean field distributin leads t a tractable lwer bund, s that the equatins F[Q] Q(h j ) = cnst (14) can be slved efficiently2. Since h(, H) in (1) is linear in the arginals Q(hjl), it fllws that its derivative with respect t Q(hj 1) is independent f Q(hjl), althugh it will be a functin f the ther arginals, and s the ptiizatin f (11) with respect t individual arginals again takes the fr (14) and by assuptin is therefre sluble. Next we cnsider the ptiizatin with respect t the ixing cefficients a. Since all f the ters in (11) are linear in a, except fr the entrpy ter, we can write F),[Qix(H)] = I:a(-E) - I:alna + 1 (15) 2In standard ean field thery the cnstant wuld be zer, but fr any dels f interest the slightly re general equatins given by (14) will again be sluble.

5 42 C. M. Bishp, N. Lawrence, T. Jaakkla and M. L Jrdan where we have used (1) and defined F[Q(Hl)] + L Q(Hl) InR(Hl) {H} + LAk LR(Hlk)Q(Hl) +lna. k {H} (16) Maxiizing (15) with respect t a, subject t the cnstraints ~ a ~ 1 and L a = 1, we see that the ixing cefficients which axiize the lwer bund are given by the Bltzann distributin exp(-e) a = Lk exp(-ek)' We next axiize the bund (11) with respect t the sthing arginals R(hj 1). Se anipulatin leads t the slutin (17) R(hil) = aqa~il) [~>'~ 'Q('k)Q(hi'k)l-1 (1) in which 7r~,Q(, k) dentes the expressin defined in (12) but with the j ter itted fr the prduct. The ptiizatin f the JLj takes the fr f a re-estiatin frula given by an extensin f the result btained fr ean-field thery by Saul et al. (1996). Fr siplicity we it the details here. Finally, we ptiize the bund with respect t the A, t give 1 1 ~ = - L 7rR,Q(, k). a k Since the varius paraeters are cupled, and we have ptiized the individually keeping the reainder cnstant, it will be necessary t axiize the lwer bund iteratively until se cnvergence criterin is satisfied. Having dne this fr a particular instantiatin f the visible ndes, we can then deterine the gradients f the bund with respect t the paraeters gverning the riginal belief netwrk, and use these gradients fr learning. 3 Applicatin t Sigid Belief Netwrks We illustrate the ixtures fralis by cnsidering its applicatin t sigid belief netwrks f the fr (2). The cpnents f the ixture distributin are given by factrized Bernulli distributins f the fr (3) with paraeters JLi. Again we have t intrduce variatinal paraeters ~i fr each cpnent using (4). The paraeters {JLi, ~i} are ptiized alng with {a, R(hjl), A} fr each pattern in the training set. We first investigate the extent t which the use f a ixture distributin yields an iprveent in the lwer bund n the lg likelihd cpared with standard ean field thery. T d this, we fllw Saul et al. (1996) and cnsider layered netwrks having 2 units in the first layer, 4 units in the secnd layer and 6 units in the third layer, with full cnnectivity between layers. In all cases the six final-layer units are cnsidered t be visible and have their states claped at zer. We generate 5 netwrks with paraeters {Jij, bi } chsen randly with unifr distributin ver (-1, 1). The nuber f hidden variable cnfiguratins is 2 6 = 64 and is sufficiently sall that the true lg likelihd can be cputed directly by suatin ver the hidden states. We can therefre cpare the value f (19)

6 Apprxiating Psterir Distributins in Belief Netwrks Using Mixtures 421 the lwer bund F with the true lg likelihd L, using the nnnalized errr (L - F)/ L. Figure 1 shws histgras f the relative lg likelihd errr fr varius nubers f ixture cpnents, tgether with the ean values taken fr the histgras. These shw a systeatic iprveent in the quality f the apprxiatin as the nuber f ixture cpnents is increased. 5 cpnents, ean: ~r-----~----~----~ cpnents, ean: r-----~----~----~--_ cpnents, ean: cpnents, ean: r-----~----~----~--_ cpnent, ean: ~r-----~----~----~--~ Gn..:-p----~----~----~ , g.14 III c as ~ '------~----~----~-----' n. f cpnents Figure 1: Plts f histgras f the nralized errr between the true lg likelihd and the lwer bund. fr varius nubers f ixture cpnents. Als shwn is the ean values taken fr the histgras. pltted against the nuber f cpnents. Next we cnsider the ipact f using ixture distributins n learning. T explre this we use a sall-scale prble intrduced by Hintn et al. (1995) invlving binary iages f size 4 x 4 in which each iage cntains either hrizntal r vertical bars with equal prbability, with each f the fur pssible lcatins fr a bar ccupied with prbability.5. We trained netwrks having architecture using distributins having between 1 and 5 cpnents. Randly generated patterns were presented t the netwrk fr a ttal f 5 presentatins, and the J-ti and ~i were initialised fr a unifnn distributin ver (,1). Again the netwrks are sufficiently sall that the exact lg likelihd fr the trained dels can be evaluated directly. A Hintn diagra f the hidden-t-utput weights fr the eight units in a netwrk trained with 5 ixture cpnents is shwn in Figure 2. Figure 3 shws a plt f the true lg likelihd versus the nuber M f cpnents in the ixture fr a set f experients in which, fr each value f M, the del was trained 1 ties starting fr different rand paraeter initializatins. These results indicate that, as the nuber f ixture cpnents is increased, the learning algrith is able t find a set f netwrk paraeters having a larger likelihd, and hence that the iprved flexibility f the apprxiating distributin is indeed translated int an iprved training algrith. We are currently applying the ixture fnnalis t the large-scale prble f hand-written digit classificatin.

7 422.. t j jo C. M. Bishp, N. Lawrence, T. Jaakkla and M. I. Jrdan -. - Figure 2: Hintn diagras f the hidden-t-utput weights fr each f the hidden units in a netwrk trained n the 'bars' prble using a ixture distributin having 5 cpnents ' : - Q) ~ CI.S! Q) -9 E -1 ~ 6 B n. f cpnents 5 6 Figure 3: True lg likelihd (divided by the nuber f patterns) versus the nuber M f ixture cpnents fr the 'bars' prble indicating a systeatic iprveent in perfrance as M is increased. References Hintn, G. E., P. Dayan, B. 1. Frey, and R. M. Neal (1995). The wake-sleep algrith fr unsupervised neural netwrks. Science 26, Jaakkla, T. (1997). Variatinal Methds fr Inference and Estiatin in Graphical Mdels. Ph.D. thesis, MIT. Jaakkla, T. and M. I. Jrdan (1997). Apprxiating psterirs via ixture dels. T appear in Prceedings NATO ASI Learning in Graphical Mdels, Ed. M. I. Jrdan. Kluwer. Neal, R. (1992). Cnnectinist learning f belief netwrks. Artificial Intelligence 56, Saul, L. K., T. Jaakkla, and M. I. Jrdan (1996). Mean field thery fr sigid belief netwrks. Jurnal f Artificial Intelligence Research 4, Saul, L. K. and M. I. Jrdan (1996). Expliting tractable substructures in intractable netwrks. In D. S. Turetzky, M. C. Mzer, and M. E. Hassel (Eds.), Advances in Neural Infratin Prcessing Systes, Vlue, pp MIT Press.