Advnced Mchne Lernng Vrtonl Inference Erc ng Lecture 12, August 12, 2009 Redng: Erc ng Erc ng @ CMU, 2006-2009 1 An Isng model on 2-D mge odes encode hdden nformton ptchdentty. They receve locl nformton from the mge rghtness, color. Informton s propgted though the grph over ts edges. Edges encode comptlty etween nodes. r or wter?? Erc ng Erc ng @ CMU, 2006-2009 2 1
Why Appromte Inference? Tree-wdth of grph s O cn e huge numer~1000s of pels Ect nference wll e too epensve 1 ep θ + θ Z < p 0 Erc ng Erc ng @ CMU, 2006-2009 3 Vrtonl Methods For dstruton p θ ssocted wth comple grph, computng the mrgnl or condtonl prolty of rtrry rndom vrles s ntrctle Vrtonl methods formultng prolstc nference s n optmzton prolem: e.g. f * rg m or mn f S { F f } f : trctle prolty dstruton or, solutons to certn prolstc queres Erc ng Erc ng @ CMU, 2006-2009 4 2
Bethe Energy Mnmzton Erc ng Erc ng @ CMU, 2006-2009 5 The Oectve Let us cll the ctul dstruton P We wsh to fnd dstruton Q such tht Q s good ppromton to P Recll the defnton of KL-dvergence KLQ 1 Q 2 >0 KLQ 1 Q 2 0 ff Q 1 Q 2 But, KLQ 1 Q 2 KLQ 2 Q 1 P 1/ Z f ff Q1 KL Q1 Q2 Q1 log Q 2 Erc ng Erc ng @ CMU, 2006-2009 6 3
Whch KL? Computng KLP Q requres nference! But KLP Q cn e computed wthout performng nference on P Q KL Q P Q log P Usng H Q logq Q log P Q E log P P 1/ Z f ff KL Q P HQ EQ log1/ Z f ff ff H log1/ Z E log f Q Q Q Erc ng Erc ng @ CMU, 2006-2009 7 The Oectve KL Q P H Q EQ log f + log Z ff F P, Q We wll cll F P, Q the Energy Functonl * F P, P? FP,Q > FP,P *lso clled Gs Free Energy Erc ng Erc ng @ CMU, 2006-2009 8 4
The Energy Functonl Let us loo t the functonl E Q log f cn e computed f we hve mrgnls over ech f ff H Q Q logq s hrder! Requres summton over ll possle vlues Computng F, s therefore hrd n generl. Approch 1: Appromte F P, Q wth esy to compute F P, Q H E log f Q Q ff F P, Q Erc ng Erc ng @ CMU, 2006-2009 9 Tree Energy Functonls Consder tree-structured dstruton 1 2 3 4 5 6 7 8 The prolty cn e wrtten s: H tree F Tree, E 1, nvolves summton over edges nd vertces nd s therefore esy to compute Erc ng Erc ng @ CMU, 2006-2009 10, ln, + 1 ln, E,, ln, + 1 ln, ln f,, ln f, E,, E,,, ln + ln 2 ln E, f, f F,, 12 + F23 +.. + F67 + F78 F1 F5 F2 F6 F3 F7 5
Tree Energy Functonls Consder tree-structured dstruton 1 2 3 4 The prolty cn e wrtten s: Htree FTree ln + d 1 ln ln + 1 d ln f F 5 6 7 8 12 + F23 +.. + F67 + F78 F1 F5 F2 F6 F3 F7 1 d nvolves summton over edges nd vertces nd s therefore esy to compute Erc ng Erc ng @ CMU, 2006-2009 11 Bethe Appromton to Gs Free Energy For generl grph, choose F P, Q F Beth H Bethe ln + d 1 ln F Bethe ln + d f Heth f 1 ln Clled Bethe ppromton fter the physcst Hns Bethe 1 2 3 4 5 6 7 8 F ethe F 2 F.. F 12 + F23 +.. + F67 + F78 F1 F5 F2 2 Equl to the ect Gs free energy when the fctor grph s tree In generl, H Bethe s not the sme s the H of tree Erc ng Erc ng @ CMU, 2006-2009 12 6 8 6
Bethe Appromton Pros: Esy to compute, snce entropy term nvolves sum over prwse nd sngle vrles Cons: F P, Q F my or my not e well connected to F P, Q ethe It could, n generl, e greter, equl or less thn F P, Q Optmze ech 's. For dscrete elef, constrned opt. wth Lgrngn multpler For contnuous elef, not yet generl formul ot lwys converge Erc ng Erc ng @ CMU, 2006-2009 13 From GM to fctored grphs ψ ψ, Prents Undrected grph Mrov rndom feld 1 P ψ ψ, Z P Drected grph Byesn networ P prents fctor grphs nterctons vrles Erc ng Erc ng @ CMU, 2006-2009 14 7
8 Erc ng Erc ng @ CMU, 2006-2009 15 Recll Belefs nd messges n FG m f elefs messges \ c c m f The elef s the BP ppromton of the mrgnl prolty. Erc ng Erc ng @ CMU, 2006-2009 16 Bethe Free Energy for FG + Beth d f F ln ln 1 + Bethe d H ln ln 1 eth Bethe H f F
9 Erc ng Erc ng @ CMU, 2006-2009 17 + + Bethe F L \ 1} { λ γ 0 L 1 1 ep d λ 0 L + ep E λ Constrned Mnmzton of the Bethe Free Energy Erc ng Erc ng @ CMU, 2006-2009 18 Bethe BP on FG Identfy to otn BP equtons: m f elefs messges \ c c m f The elef s the BP ppromton of the mrgnl prolty. m ln λ
10 Erc ng Erc ng @ CMU, 2006-2009 19 Usng, \ we get m f m \ \ \ BP Messge-updte Rules A sum product lgorthm Erc ng Erc ng @ CMU, 2006-2009 20 M M M, ψ ψ Comptltes nterctons eternl evdence M ψ Belef Propgton on trees BP Messge-updte Rules BP on trees lwys converges to ect mrgnls cf. Juncton tree lgorthm
Belef Propgton on loopy grphs M BP Messge-updte Rules M ψ, ψ M My not converge or converge to wrong soluton eternl evdence Comptltes nterctons ψ M Erc ng Erc ng @ CMU, 2006-2009 21 Loopy Belef Propgton If BP s used on grphs wth loops, messges my crculte ndefntely Emprclly, good ppromton s stll chevle Stop fter fed # of tertons Stop when no sgnfcnt chnge n elefs If soluton s not osclltory ut converges, t usully s good ppromton Erc ng Erc ng @ CMU, 2006-2009 22 11
The Theory Behnd LBP For dstruton p θ ssocted wth comple grph, computng the mrgnl or condtonl prolty of rtrry rndom vrles s ntrctle Vrtonl methods formultng prolstc nference s n optmzton prolem: q * rg mn qs { F p, q } Beth F Bethe ln + 1 f d ln f Hethe q : trctle prolty dstruton Erc ng Erc ng @ CMU, 2006-2009 23 The Theory Behnd LBP But we do not optmze q eplctly, focus on the set of elefs e.g., {, τ,, τ } Rel the optmzton prolem ppromte oectve: reled fesle set: rg mn The loopy BP lgorthm: * M { E + F } fed pont terton procedure tht tres to solve * o H q F,, M{ τ 0 τ M 1, M τ, τ } H H M o Beth M o o Erc ng Erc ng @ CMU, 2006-2009 24 12
Men Feld Appromton Erc ng Erc ng @ CMU, 2006-2009 25 Men feld methods Optmze q H n the spce of trctle fmles.e., sugrph of G p over whch ect computton of H q s fesle Tghtenng the optmzton spce ect oectve: tghtened fesle set: H q Q T T Q q * rg mn qt E q H q Erc ng Erc ng @ CMU, 2006-2009 26 13
Cluster-sed ppro. to the Gs free energy Wegernc 2001, ng et l 03,04 Ect: G[ p ] Clusters: G[{ q c c}] ntrctle Erc ng Erc ng @ CMU, 2006-2009 27 Men feld ppro. to Gs free energy Gven dsont clusterng, {C 1,, C I }, of ll vrles Let q, Men-feld free energy G MF Wll never equl to the ect Gs free energy no mtter wht clusterng s used, ut t does lwys defne lower ound of the lelhood Optmze ech q c 's. Vrtonl clculus q C q C E C + q C ln q C C Do nference n ech q c usng ny trctle lgorthm C q q φ + q + q ln q e.g., GMF φ < nïve men feld Erc ng Erc ng @ CMU, 2006-2009 28 14
The Generlzed Men Feld theorem Theorem: The optmum GMF ppromton to the cluster mrgnl s somorphc to the cluster posteror of the orgnl dstruton gven nternl evdence nd ts generlzed men felds: * q H, C p H, C E, C, H, MB q GMF lgorthm: Iterte over ech q Erc ng Erc ng @ CMU, 2006-2009 29 A generlzed men feld lgorthm [ng et l. UAI 2003] Erc ng Erc ng @ CMU, 2006-2009 30 15
A generlzed men feld lgorthm [ng et l. UAI 2003] Erc ng Erc ng @ CMU, 2006-2009 31 Convergence theorem Theorem: The GMF lgorthm s gurnteed to converge to locl optmum, nd provdes lower ound for the lelhood of evdence or prtton functon the model. Erc ng Erc ng @ CMU, 2006-2009 32 16
The nve men feld ppromton Appromte p y fully fctorzed qp q For Boltzmnn dstruton pep{ < q +q o }/Z : men Gs feld predctve equton: dstruton: pq ep ep θ 0 θ 0 + + θ θ + + A q p p { { : : } }} { qq I q q resemles messge sent from node to { q : : } } forms the men feld ppled to from ts neghorhood { q Erc ng Erc ng @ CMU, 2006-2009 33 Generlzed MF ppromton to Isng models Cluster mrgnl of squre loc C : q C ep θ + + θ0 θ, C C C, MB, ' MBC q C ' Vrtully reprmeterzed Isng model of smll sze. Erc ng Erc ng @ CMU, 2006-2009 34 17
GMF ppromton to Isng models GMF 44 GMF 22 BP Attrctve couplng: postvely weghted Repulsve couplng: negtvely weghted Erc ng Erc ng @ CMU, 2006-2009 35 Automtc Vrtonl Inference S 1 S 2 S 3... S y 11... y 1 y 12 y 13...... y 2... y 3... y 1... y A 1 A 2 A 3... A fhmm Men feld ppro. Structured vrtonl ppro. Currently for ech new model we hve to derve the vrtonl updte equtons wrte pplcton-specfc code to fnd the soluton Ech cn e tme consumng nd error prone Cn we uld generl-purpose nference engne whch utomtes these procedures? Erc ng Erc ng @ CMU, 2006-2009 36 18
Cluster-sed MF e.g., GMF generl, tertve messge pssng lgorthm clusterng completely defnes ppromton preserves dependences flele performnce/cost trde-off clusterng utomtle recovers model-specfc structured VI lgorthms, ncludng: fhmm, LDA vrtonl Byesn lernng lgorthms esly provdes new structured VI ppromtons to comple models Erc ng Erc ng @ CMU, 2006-2009 37 Emple: Fctorl HMM Erc ng Erc ng @ CMU, 2006-2009 38 19