From Elimination to Belief Propagation

Size: px

Start display at page:

Download "From Elimination to Belief Propagation"

Sophia Foster
5 years ago
Views:

1 School of omputr Scinc Th lif Propagation (Sum-Product lgorithm Probabilistic Graphical Modls ( Lctur 5, Sp 31, 2007 Rcptor Kinas Rcptor Kinas Kinas X 5 ric Xing Gn G T X 6 X 7 Gn H X 8 Rading: J-hap 4 1 rom limination to lif Propagation Rcall that Inducd dpndncy during marginalization is capturd in limination cliqus Summation <-> limination Intrmdiat trm <-> limination cliqu an this lad to an gnric infrnc algorithm? G H ric Xing 2 1

2 Tr GMs Undirctd tr: a uniqu path btwn any pair of nods irctd tr: all nods xcpt th root hav xactly on parnt Poly tr: can hav multipl parnts W will com back to this latr ric Xing 3 quivalnc of dirctd and undirctd trs ny undirctd tr can b convrtd to a dirctd tr by choosing a root nod and dircting all dgs away from it dirctd tr and th corrsponding undirctd tr mak th sam conditional indpndnc assrtions Paramtrizations ar ssntially th sam. Undirctd tr: irctd tr: quivalnc: vidnc:? ric Xing 4 2

3 rom limination to mssag passing Rcall LIMINTION algorithm: hoos an ordring Z in which qury nod f is th final nod Plac all potntials on an activ list liminat nod i by rmoving all potntials containing i, tak sum/product ovr x i. Plac th rsultant factor back on th list or a TR graph: hoos qury nod f as th root of th tr Viw tr as a dirctd tr with dgs pointing towards from f limination ordring basd on dpth-first travrsal limination of ach nod can b considrd as mssag-passing (or lif Propagation dirctly along tr branchs, rathr than on som transformd graphs thus, w can us th tr itslf as a data-structur to do gnral infrnc!! ric Xing 5 Th limination algorithm Procdur Initializ (G, Z 1. Lt Z 1,...,Z k b an ordring of Z such that Z i Z j iff i < j 2. Initializ with th full th st of factors Procdur vidnc ( 1. for ach i Ι, = δ( i, i Procdur Sum-Product-Variabl- limination (, Z, 1. for i = 1,..., k Sum-Product-liminat-Var(, Z i 2. φ φ φ 3. rturn φ 4. Normalization (φ Procdur Normalization (φ 1. P(X =φ (X/ x φ (X Procdur Sum-Product-liminat-Var (, // St of factors Z // Variabl to b liminatd 1. {φ : Z Scop[φ]} ψ φ φ 4. τ Z ψ 5. rturn {τ} ric Xing 6 3

4 Mssag passing for trs f Lt m ij (x i dnot th factor rsulting from liminating variabls from bllow up to i, which is a function of x i : i This is rminiscnt of a mssag snt from j to i. j k l m ij (x i rprsnts a "blif" of x i from x j! ric Xing 7 limination on trs is quivalnt to mssag passing along tr branchs! f i j k l ric Xing 8 4

5 Th mssag passing protocol: nod can snd a mssag to its nighbors whn (and only whn it has rcivd mssags from all its othr nighbors. omputing nod marginals: Naïv approach: considr ach nod as th root and xcut th mssag passing algorithm m 21 (x 1 omputing P( m 32 (x 2 m 42 (x 2 ric Xing 9 Th mssag passing protocol: nod can snd a mssag to its nighbors whn (and only whn it has rcivd mssags from all its othr nighbors. omputing nod marginals: Naïv approach: considr ach nod as th root and xcut th mssag passing algorithm m 12 (x 2 omputing P( m 32 (x 2 m 42 (x 2 ric Xing 10 5

6 Th mssag passing protocol: nod can snd a mssag to its nighbors whn (and only whn it has rcivd mssags from all its othr nighbors. omputing nod marginals: Naïv approach: considr ach nod as th root and xcut th mssag passing algorithm m 12 (x 2 omputing P( m 23 (x 3 m 42 (x 2 ric Xing 11 omputing nod marginals Naïv approach: omplxity: N N is th numbr of nods is th complxity of a complt mssag passing ltrnativ dynamic programming approach 2-Pass algorithm (nxt slid omplxity: 2! ric Xing 12 6

7 Th mssag passing protocol: two-pass algorithm: m 21 ( m 12 ( m 32 ( m 42 ( m 24 ( m 23 ( ric Xing 13 lif Propagation (SP-algorithm: Squntial implmntation ric Xing 14 7

8 lif Propagation (SP-algorithm: Paralll synchronous implmntation or a nod of dgr d, whnvr mssags hav arrivd on any subst of d-1 nod, comput th mssag for th rmaining dg and snd! pair of mssags hav bn computd for ach dg, on for ach dirction ll incoming mssags ar vntually computd for ach nod ric Xing 15 orrctnss of P on tr ollollary: th synchronous implmntation is "non-blocking" Thm: Th Mssag Passag Guarants obtaining all marginals in th tr What about non-tr? ric Xing 16 8

9 nothr viw of SP: actor Graph xampl 1 X 5 f a f d X 5 f c f b f P( P( P(, P(X 5, P(, f a ( f b ( f c (,, f d (X 5,, f (,, ric Xing 17 actor Graphs xampl 2 f a f c ψ(x 1,x 2,x 3 = f a (x 1,x 2 f b (x 2,x 3 f c (x 3,x 1 f b xampl 3 f a ψ(x 1,x 2,x 3 = f a (x 1,x 2,x 3 ric Xing 18 9

10 actor Tr actor graph is a actor Tr if th undirctd graph obtaind by ignoring th distinction btwn variabl nods and factor nods is an undirctd tr f a ψ(x 1,x 2,x 3 = f a (x 1,x 2,x 3 ric Xing 19 Mssag Passing on a actor Tr Two kinds of mssags 1. ν: from variabls to factors 2. µ: from factors to variabls f s x i f 1 x j x i f s f 3 x k ric Xing 20 10

11 Mssag Passing on a actor Tr, con'd Mssag passing protocol: nod can snd a mssag to a nighboring nod only whn it has rcivd mssags from all its othr nighbors Marginal probability of nods: f s x i f 1 x j x i f s f 3 x k P(x i s 2 N(i µ si (x i ν is (x i µ si (x i ric Xing 21 P on a actor Tr ν 1d µ d2 µ 2 ν 3 f d µ d1 ν 2d ν 2 µ 3 µ µ a1 c3 µb2 ν 1a ν 2b f ν 3c f a f b f c ric Xing 22 11

12 X 5 X 6 Why factor graph? Tr-lik graphs to actor trs X 5 X 6 ric Xing 23 Poly-trs to actor trs X 5 X 5 ric Xing 24 12

13 X 5 X 6 Why factor graph? caus G turns tr-lik graphs to factor trs, and trs ar a data-structur that guarants corrctnss of P! X 5 X 6 X 5 X 5 ric Xing 25 Max-product algorithm: computing MP probabilitis f i j k l ric Xing 26 13

14 Max-product algorithm: computing MP configurations using a final bookkping backward pass f i j k l ric Xing 27 Summary Sum-Product algorithm computs singlton marginal probabilitis on: Trs Tr-lik graphs Poly-trs Maximum a postriori configurations can b computd by rplacing sum with max in th sum-product algorithm xtra bookkping rquird ric Xing 28 14

15 Infrnc on gnral GM Now, what if th GM is not a tr-lik graph? an w still dirctly run mssag mssag-passing protocol along its dgs? or non-trs, w do not hav th guarant that mssag-passing will b consistnt! Thn what? onstruct a graph data-structur from P that has a tr structur, and run mssag-passing on it! Junction tr algorithm ric Xing 29 limination liqu Rcall that Inducd dpndncy during marginalization is capturd in limination cliqus Summation <-> limination Intrmdiat trm <-> limination cliqu an this lad to an gnric infrnc algorithm? G H ric Xing 30 15

16 16 ric Xing 31 liqu Tr H G m h m g m m f m b m c m d = f g a m m d c p d c a m, ( (, (,, ( ric Xing 32 limination mssag passing on a cliqu tr Mssags can b rusd H G m h m g m m f m b m c m d G H G H G rom limination to Mssag Passing = f g a m m d c p d c a m, ( (, (,, (

17 rom limination to Mssag Passing limination mssag passing on a cliqu tr nothr qury... m d mc m m b m f G m g H m h Mssags m f and m h ar rusd, othrs nd to b rcomputd ric Xing 33 17

Searching Linked Lists. Perfect Skip List. Building a Skip List. Skip List Analysis (1) Assume the list is sorted, but is stored in a linked list.

Searching Linked Lists. Perfect Skip List. Building a Skip List. Skip List Analysis (1) Assume the list is sorted, but is stored in a linked list. 3 3 4 8 6 3 3 4 8 6 3 3 4 8 6 () (d) 3 Sarching Linkd Lists Sarching Linkd Lists Sarching Linkd Lists ssum th list is sortd, but is stord in a linkd list. an w us binary sarch? omparisons? Work? What if