Advanced Machine Learning. An Ising model on 2-D image

Transcription

1 Advnced Mchne Lernng Vrtonl Inference Erc ng Lecture 12, August 12, 2009 Redng: Erc ng Erc CMU, An Isng model on 2-D mge odes encode hdden nformton ptchdentty. They receve locl nformton from the mge rghtness, color. Informton s propgted though the grph over ts edges. Edges encode comptlty etween nodes. r or wter?? Erc ng Erc CMU,

2 Why Appromte Inference? Tree-wdth of grph s O cn e huge numer~1000s of pels Ect nference wll e too epensve 1 ep θ + θ Z < p 0 Erc ng Erc CMU, Vrtonl Methods For dstruton p θ ssocted wth comple grph, computng the mrgnl or condtonl prolty of rtrry rndom vrles s ntrctle Vrtonl methods formultng prolstc nference s n optmzton prolem: e.g. f * rg m or mn f S { F f } f : trctle prolty dstruton or, solutons to certn prolstc queres Erc ng Erc CMU,

3 Bethe Energy Mnmzton Erc ng Erc CMU, The Oectve Let us cll the ctul dstruton P We wsh to fnd dstruton Q such tht Q s good ppromton to P Recll the defnton of KL-dvergence KLQ 1 Q 2 >0 KLQ 1 Q 2 0 ff Q 1 Q 2 But, KLQ 1 Q 2 KLQ 2 Q 1 P 1/ Z f ff Q1 KL Q1 Q2 Q1 log Q 2 Erc ng Erc CMU,

4 Whch KL? Computng KLP Q requres nference! But KLP Q cn e computed wthout performng nference on P Q KL Q P Q log P Usng H Q logq Q log P Q E log P P 1/ Z f ff KL Q P HQ EQ log1/ Z f ff ff H log1/ Z E log f Q Q Q Erc ng Erc CMU, The Oectve KL Q P H Q EQ log f + log Z ff F P, Q We wll cll F P, Q the Energy Functonl * F P, P? FP,Q > FP,P *lso clled Gs Free Energy Erc ng Erc CMU,

5 The Energy Functonl Let us loo t the functonl E Q log f cn e computed f we hve mrgnls over ech f ff H Q Q logq s hrder! Requres summton over ll possle vlues Computng F, s therefore hrd n generl. Approch 1: Appromte F P, Q wth esy to compute F P, Q H E log f Q Q ff F P, Q Erc ng Erc CMU, Tree Energy Functonls Consder tree-structured dstruton The prolty cn e wrtten s: H tree F Tree, E 1, nvolves summton over edges nd vertces nd s therefore esy to compute Erc ng Erc CMU, , ln, + 1 ln, E,, ln, + 1 ln, ln f,, ln f, E,, E,,, ln + ln 2 ln E, f, f F,, 12 + F F67 + F78 F1 F5 F2 F6 F3 F7 5

6 Tree Energy Functonls Consder tree-structured dstruton The prolty cn e wrtten s: Htree FTree ln + d 1 ln ln + 1 d ln f F F F67 + F78 F1 F5 F2 F6 F3 F7 1 d nvolves summton over edges nd vertces nd s therefore esy to compute Erc ng Erc CMU, Bethe Appromton to Gs Free Energy For generl grph, choose F P, Q F Beth H Bethe ln + d 1 ln F Bethe ln + d f Heth f 1 ln Clled Bethe ppromton fter the physcst Hns Bethe F ethe F 2 F.. F 12 + F F67 + F78 F1 F5 F2 2 Equl to the ect Gs free energy when the fctor grph s tree In generl, H Bethe s not the sme s the H of tree Erc ng Erc CMU,

7 Bethe Appromton Pros: Esy to compute, snce entropy term nvolves sum over prwse nd sngle vrles Cons: F P, Q F my or my not e well connected to F P, Q ethe It could, n generl, e greter, equl or less thn F P, Q Optmze ech 's. For dscrete elef, constrned opt. wth Lgrngn multpler For contnuous elef, not yet generl formul ot lwys converge Erc ng Erc CMU, From GM to fctored grphs ψ ψ, Prents Undrected grph Mrov rndom feld 1 P ψ ψ, Z P Drected grph Byesn networ P prents fctor grphs nterctons vrles Erc ng Erc CMU,

8 8 Erc ng Erc CMU, Recll Belefs nd messges n FG m f elefs messges \ c c m f The elef s the BP ppromton of the mrgnl prolty. Erc ng Erc CMU, Bethe Free Energy for FG + Beth d f F ln ln 1 + Bethe d H ln ln 1 eth Bethe H f F

9 9 Erc ng Erc CMU, Bethe F L \ 1} { λ γ 0 L 1 1 ep d λ 0 L + ep E λ Constrned Mnmzton of the Bethe Free Energy Erc ng Erc CMU, Bethe BP on FG Identfy to otn BP equtons: m f elefs messges \ c c m f The elef s the BP ppromton of the mrgnl prolty. m ln λ

10 10 Erc ng Erc CMU, Usng, \ we get m f m \ \ \ BP Messge-updte Rules A sum product lgorthm Erc ng Erc CMU, M M M, ψ ψ Comptltes nterctons eternl evdence M ψ Belef Propgton on trees BP Messge-updte Rules BP on trees lwys converges to ect mrgnls cf. Juncton tree lgorthm

11 Belef Propgton on loopy grphs M BP Messge-updte Rules M ψ, ψ M My not converge or converge to wrong soluton eternl evdence Comptltes nterctons ψ M Erc ng Erc CMU, Loopy Belef Propgton If BP s used on grphs wth loops, messges my crculte ndefntely Emprclly, good ppromton s stll chevle Stop fter fed # of tertons Stop when no sgnfcnt chnge n elefs If soluton s not osclltory ut converges, t usully s good ppromton Erc ng Erc CMU,

12 The Theory Behnd LBP For dstruton p θ ssocted wth comple grph, computng the mrgnl or condtonl prolty of rtrry rndom vrles s ntrctle Vrtonl methods formultng prolstc nference s n optmzton prolem: q * rg mn qs { F p, q } Beth F Bethe ln + 1 f d ln f Hethe q : trctle prolty dstruton Erc ng Erc CMU, The Theory Behnd LBP But we do not optmze q eplctly, focus on the set of elefs e.g., {, τ,, τ } Rel the optmzton prolem ppromte oectve: reled fesle set: rg mn The loopy BP lgorthm: * M { E + F } fed pont terton procedure tht tres to solve * o H q F,, M{ τ 0 τ M 1, M τ, τ } H H M o Beth M o o Erc ng Erc CMU,

13 Men Feld Appromton Erc ng Erc CMU, Men feld methods Optmze q H n the spce of trctle fmles.e., sugrph of G p over whch ect computton of H q s fesle Tghtenng the optmzton spce ect oectve: tghtened fesle set: H q Q T T Q q * rg mn qt E q H q Erc ng Erc CMU,

14 Cluster-sed ppro. to the Gs free energy Wegernc 2001, ng et l 03,04 Ect: G[ p ] Clusters: G[{ q c c}] ntrctle Erc ng Erc CMU, Men feld ppro. to Gs free energy Gven dsont clusterng, {C 1,, C I }, of ll vrles Let q, Men-feld free energy G MF Wll never equl to the ect Gs free energy no mtter wht clusterng s used, ut t does lwys defne lower ound of the lelhood Optmze ech q c 's. Vrtonl clculus q C q C E C + q C ln q C C Do nference n ech q c usng ny trctle lgorthm C q q φ + q + q ln q e.g., GMF φ < nïve men feld Erc ng Erc CMU,

15 The Generlzed Men Feld theorem Theorem: The optmum GMF ppromton to the cluster mrgnl s somorphc to the cluster posteror of the orgnl dstruton gven nternl evdence nd ts generlzed men felds: * q H, C p H, C E, C, H, MB q GMF lgorthm: Iterte over ech q Erc ng Erc CMU, A generlzed men feld lgorthm [ng et l. UAI 2003] Erc ng Erc CMU,

16 A generlzed men feld lgorthm [ng et l. UAI 2003] Erc ng Erc CMU, Convergence theorem Theorem: The GMF lgorthm s gurnteed to converge to locl optmum, nd provdes lower ound for the lelhood of evdence or prtton functon the model. Erc ng Erc CMU,

17 The nve men feld ppromton Appromte p y fully fctorzed qp q For Boltzmnn dstruton pep{ < q +q o }/Z : men Gs feld predctve equton: dstruton: pq ep ep θ 0 θ θ θ + + A q p p { { : : } }} { qq I q q resemles messge sent from node to { q : : } } forms the men feld ppled to from ts neghorhood { q Erc ng Erc CMU, Generlzed MF ppromton to Isng models Cluster mrgnl of squre loc C : q C ep θ + + θ0 θ, C C C, MB, ' MBC q C ' Vrtully reprmeterzed Isng model of smll sze. Erc ng Erc CMU,

18 GMF ppromton to Isng models GMF 44 GMF 22 BP Attrctve couplng: postvely weghted Repulsve couplng: negtvely weghted Erc ng Erc CMU, Automtc Vrtonl Inference S 1 S 2 S 3... S y y 1 y 12 y y 2... y 3... y 1... y A 1 A 2 A 3... A fhmm Men feld ppro. Structured vrtonl ppro. Currently for ech new model we hve to derve the vrtonl updte equtons wrte pplcton-specfc code to fnd the soluton Ech cn e tme consumng nd error prone Cn we uld generl-purpose nference engne whch utomtes these procedures? Erc ng Erc CMU,

19 Cluster-sed MF e.g., GMF generl, tertve messge pssng lgorthm clusterng completely defnes ppromton preserves dependences flele performnce/cost trde-off clusterng utomtle recovers model-specfc structured VI lgorthms, ncludng: fhmm, LDA vrtonl Byesn lernng lgorthms esly provdes new structured VI ppromtons to comple models Erc ng Erc CMU, Emple: Fctorl HMM Erc ng Erc CMU,