Constructing Free Energy Approximations and GBP Algorithms

Transcription

1 3710 Advnced Topcs n A ecture 15 Brnslv Kveton kveton@cs.ptt.edu 5802 ennott qure onstructng Free Energy Approxtons nd BP Algorths ontent Why? Belef propgton (BP) Fctor grphs egon-sed free energy pproxtons Bethe ethod Bethe ethod nd BP egon grphs enerlzed elef propgton (BP)

2 BP on luster Trees, τ(),, τ(,),, τ(,) τ(,) Belef propgton on,,, cluster trees s n exct,, ethod for coputng posteror rgnls pce coplexty s exponentl n the treewdth of the grph BP on luster rphs, τ(),, τ(,),, τ() τ() τ() Belef propgton on,,, cluster grphs s n τ() pproxte ethods,, for coputng posteror rgnls τ() onvergence s not gurnteed! pce coplexty s lner n the sze of the lrgest cluster

3 Fctor rphs,,,,,,,, A fctor grph s prtte grph tht represents fctored structure,, luster tree Vrle nodes c d e Fctor grph Fctor nodes Fctor rphs,,,,,,,, A fctor grph s prtte grph tht represents fctored structure,, luster tree Vrle nodes c d e Fctor grph Fctor nodes

4 BP on Fctor rphs Messges: Mrgnls: n ( x ) ( x ) c N () \ c ( x ) ( x ) ( x ) f ( x ) n j ( x j ) N () ( ) x \ x j N \ c d e BP on Fctor rphs Messges: Mrgnls: n ( x ) ( x ) c N () \ c ( x ) ( x ) ( x ) f ( x ) n j ( x j ) N () ( ) x \ x j N \ P(x h )? c d e

5 BP on Fctor rphs Messges: Mrgnls: n ( x ) ( x ) c N () \ c ( x ) ( x ) ( x ) f ( x ) n j ( x j ) N () ( ) x \ x j N \ P(x h )? c d e BP on Fctor rphs Messges: Mrgnls: n ( x ) ( x ) c N () \ c ( x ) ( x ) ( x ) f ( x ) n j ( x j ) N () ( ) x \ x j N \ P(x h )? c d e

6 BP on Fctor rphs Messges: Mrgnls: n ( x ) ( x ) c N () \ c ( x ) ( x ) ( x ) f ( x ) n j ( x j ) N () ( ) x \ x j N \ P(x h )? c d e Fctored jont: p Energy equtons: Free Energy ( x) exp [ E( x) ] Z E( x) ln ( x ) Z x exp [ E( x) ] ( ) F ( p) F + F M 1 F ln Z ( ) U ( ) ( ) U ( ) ( x) E( x) s free energy F ( ) F elholtz free energy x f ( ) ( x) ln ( x) x Vrtonl verge energy Vrtonl entropy

7 egon-bsed Free Energy A regon of fctor grph s gven y vrle set V nd fctor set A such tht f fctor node elongs to A, ll ts vrle nodes re n V The forls cn express oth Kkuch nd Bethe pproxtons A B A B nvld regon Vld regon egon-bsed Free Energy egon verge energy nd entropy: F E ( ) U ( ) ( ) ( x ) ln f ( x ) A egon-sed verge energy nd pproxte entropy: F ({ }) U ({ }) ({ }) where c s countng nuer of the regon U U ( ) ( x ) E ( x ) x ( ) ( x ) ln ( x ) x ({ }) c U ( ) ({ }) c ( )

8 egon-bsed Free Energy A regon-sed pproxton s vld f for every fctor node nd every vrle node : c A ( ) cv ( ) 1 f (x ) p (x ), then the verge energy U of vld regon-sed pproxton s exct f p(x) s unfor nd (x ) p (x ), the entropy of vld regon-sed pproxton s exct Nether fctors nor vrles re doule counted Mnzton of F cn e cheved y nzng ( p) or xzng onstrned egon-bsed Free Energy A regon-sed pproxton s constrned f: Every (x ) hs the for of prolty functon Mrgnls re consstent cross regons A constrned regon-sed pproxton s xent-norl f t s vld nd the entropy cheves ts xu when ll (x ) re unfor Mnzton of F cn e cheved y xzng espte these restrctons we y get strnge lookng results!

9 Bethe Method Bethe pproxton s specl cse of regon-sed free energy pproxton Bethe free energy equls to F Bethe U Bethe Bethe : U Bethe ( ) ( x ) ln f ( x ) f the fctor grph hs no cycles, U Bethe nd Bethe re exct Bethe pproxtons re xent-norl M 1 x M N Bethe 1 x 1 x ( ) ( x ) ln ( x ) + ( d 1) ( x ) ln ( x ) A B Bethe Method nd BP The prole of fndng F Bethe cn e stted s: nze suject to : F Bethe x x x \ x ( x ) ( x ) 1 1 ( x ) ( x ) To solve ths constrned optzton prole, we ntroduce grngn ultplers, nd turn t nto unconstrned one The frst dervtves wth respect to elefs yeld sttonry ponts, whch re fxed ponts of the BP lgorth

10 egon rphs egon grph s grphcl forls for genertng regonsed free energy pproxtons A,,1,2,4,5 2 B,,2,3,5,6,4,5 5 5,6 6,E,4,5,7,8 8 F,5,6,8,9 egon rphs egon grph s grphcl forls for genertng regonsed free energy pproxtons A,,1,2,4,5 2 B,,2,3,5,6,4,5 5 5,6 6,E,4,5,7,8 8 F,5,6,8,9 egons nd e cn e connected y n edge e only f e

11 egon rphs egon grph s grphcl forls for genertng regonsed free energy pproxtons A,,1,2,4,5,4,5,E,4,5,7,8 2 B,,2,3,5, , F,5,6,8, The countng nuer equls to one nus the n-degree egon rphs egon grph s grphcl forls for genertng regonsed free energy pproxtons A,,1,2,4,5,4,5,E,4,5,7,8 2 B,,2,3,5, , F,5,6,8, The grph hs to stsfy the regon grph condton

12 enerlzed Belef Propgton A clss of essge-pssng lgorths Prent-to-chld lgorth enerlzes the BP lgorth on regon grphs ( x ) xp\ FP \ P f ( x ) ( x ) (, ) ( P, ) ( x ) f ( x ) ( x ) ( ) (, ) N ( P, ) ( x ) P orrectness cn e proved slrly to the BP lgorth ( ) A P P P P ( ) \ε ( ) P ( x )