School o Coputer Scence Approte Inerence: Loopy Bele Propgton nd vrnts Prolstc Grphcl Models 0-708 Lecture 4, ov 7, 007 Receptor A Knse C Gene G Receptor B Knse D Knse E 3 4 5 TF F 6 Gene H 7 8 Hetunndn Ksetty Redng: J-Chp. 5,6, KF-Chp. 8 An Isng odel on -D ge odes encode hdden norton ptchdentty. They receve locl norton ro the ge rghtness, color. Inorton s propgted though the grph over ts edges. Edges encode coptlty etween nodes. r or wter?? Erc ng
Why Approte Inerence? Why cn t we ust run uncton tree on ths grph? I grd, tree wdth tlest I ~O000, we hve clque wth 00 entres Erc ng 3 A recp: Fctor Grphs ψ ψ, Prents Undrected grph Drected grph Mrov rndo eld Byesn networ P = ψ ψ, Z P = P prents ctor grphs P = / Z F nterctons vrles Erc ng 4
3 Erc ng 5 M M M, ψ ψ Coptltes nterctons eternl evdence M ψ Bele Propgton BP Messge-updte Rules BP on trees lwys converges to ect rgnls c. Juncton tree lgorth Erc ng 6 Beles nd essges n FG eles essges = \ c c = \ \
Approte Inerence: Wht to pprote? Let us cll the ctul dstruton P We wsh to nd dstruton such tht s good pproton Recll the denton o KL-dvergence KL >=0 KL KL =0 = We cn thereore use KL s scorng uncton to decde good But, KL KL P = / Z = F log Erc ng 7 Whch KL? Coputng KLP requres nerence! But KL P cn e coputed wthout perorng nerence on P KL P = log P Usng = = H log log P E log P P = / Z F KL P = H E log/ Z F F = H log/ Z E log Erc ng 8 4
Optzton uncton KL P = H E log + log Z F F P, We wll cll F P, the Free energy * F P, P =? FP, >= FP,P *Gs Free Energy Erc ng 9 The Free Energy Let us loo t the ree energy E log cn e coputed we hve rgnls over ech F F P, = H E log F = H log s hrder! Requres suton over ll possle vlues Coputng F, s thereore hrd n generl. Approch : Approte F P, wth esy to copute F P, Erc ng 0 5
Esy ree energes Consder tree-structured dstruton 3 4 5 6 7 8 d The prolty cn e wrtten s: = Htree = log + d log FTree = log + d log = F + F3 +.. + F67 + F78 F F5 F F6 F3 F7 nvolves suton over edges nd vertces nd s thereore esy to copute Erc ng Bethe Approton to Gs Free Energy For generl grph, choose F P, = log + d log Clled Bethe pproton ter the physcst Hns Bethe Equl to the ect Gs ree energy when the ctor grph s tree ote: Ths s not the se s the entropy o tree 3 4 F ethe = F 5 6 7 8 + F3 + + F67 + F78 F F5 F F6.... F 8 Erc ng 6
Bethe Approton Pros: Esy to copute, snce entropy ter nvolves su over prwse nd sngle vrles Cons: F P, = F y or y not e well connected to F P, ethe It could, n generl, e greter, equl or less thn F P, Optze ech 's. For dscrete ele, constrned opt. wth Lgrngn ultpler For contnuous ele, not yet generl orul ot lwys converge Erc ng 3 Mnzng the Bethe Free Energy L = F + + Set dervtve to zero Bethe γ { λ } \ Erc ng 4 7
Proo Erc ng 5 Erc ng 6 8
9 Erc ng 7 = = log log λ \ Bethe = BP We hd Identy to otn BP equtons: ep d λ + log ep λ Erc ng 8 A ed pont terton procedure tht tres to nze F ethe Strt wth rndo ntlzton o essges nd eles Whle not converged do At convergence, sttonrty propertes re gurnteed However, not gurnteed to converge! Loopy Bele Propgton = c c new \ = new \ \
0 Erc ng 9 Regon grphs It wll e useul to loo eplctly t the essges eng pssed Messges ro vrle to ctors Messges ro ctors to vrles Let us represent ths grphclly 3 4 5 6 7 8 Erc ng 0 Sury so r = F Z P / = F E H P F log, + = d P F log log, ep d λ + log ep λ
Better pprotons? Recll tht Bethe pproton ws F ethe F 78 = F + F 5 3 F F F +.. + F 67 F.. F 6 + 8 3 4 5 6 7 8 We could construct gger regons Rules: I regon ncludes ctor, t ust nclude the vertces s well Ech ctor nd verte ust pper n tlest one regon Assocte weght wth ech regon so tht ech verte nd ctor s counted ectly once Erc ng 3 4 5 6 7 8 ^ F = F F 6 56 F 36 + F 367 + F 6 3478 + F + F + F + F 3 7 Other regons? Erc ng
Regon-sed Approtons to the Gs Free Energy Kuch, 95 Ect: G[ q ] Regons: G[{ r r }] ntrctle Erc ng 3 Generlzed Bele Propgton Bele n regon s the product o: Locl norton ctors n regon Messges ro prent regons Messges nto descendnt regons ro prents who re not descendnts. Messge-updte rules otned y enorcng rgnlzton constrnts. Erc ng 4
Generlzed Bele Propgton 45 356 4578 5689 5 45 56 58 3 4 5 6 5 7 8 9 Erc ng 5 Generlzed Bele Propgton 45 356 4578 5689 5 45 56 58 3 4 5 6 7 8 9 5 5 5 45 65 85 Erc ng 6 3
Generlzed Bele Propgton 45 356 4578 5689 5 45 56 58 3 4 5 6 7 8 9 5 [ 45][ 45784556 58 5] 45 Erc ng 7 Generlzed Bele Propgton 45 356 4578 5689 5 45 56 58 3 4 5 6 7 8 9 5 [ ] 45 ] 45 4 5 [ 365 7845 65 85 Erc ng 8 4
Soe results Erc ng 9 Sury We dened n oectve uncton F or pprote nerence However, we ound tht optzng ths uncton ws hrd We rst pproted oectve uncton F to spler F ethe Mn o F ethe turned out to e ed ponts o BP Then we etended ths to ore coplcted pprotons The resultng lgorths coe under ly clled Generlzed Bele Propgton et clss, we wll cover other ethods o pprotons Erc ng 30 5