Introduction to Graphical Models

Transcription

1 Introution to Grhil Moels Kenji Fukumizu The Institute of Sttistil Mthemtis Comuttionl Methoology in Sttistil Inferene II

2 Introution n Review 2

3 Grhil Moels Rough Sketh Grhil moels Grh: G V E V: the set of noes E: the set of eges In grhil moels the rnom vriles re reresente y the noes. sttistil reltionshis etween the vriles re reresente y the eges. A A A B C B C B C D Direte grh D Unirete grh D Ftor grh

4 Purose of using Grhil Moels Intuitive n visul reresenttion A grh is n intuitive wy of reresenting n visulizing the reltionshis mong vriles. Ineenene / onitionl ineenene A grh reresents onitionl ineenene reltionshis mong vriles. Cusl reltionshis eision mking ignosis system et. Effiient omuttion With grhs effiient rogtion lgorithms n e efine. Belief-rogtion juntion tree lgorithm Whih rts of the moeling lok effiient omuttion? 4

5 Ineenene For simliity it is ssume tht the istriution of rnom vrile hs the roility ensity funtion x. Ineenene n Y re ineenent Y x y x y Y Y Y Dwi s nottion 5

6 Conitionl Ineenene Conitionl roility Conitionl roility ensity of Y given Conitionl ineenene Def. Y y x n Y re onitionlly ineenent given Z ef. Y x y x y Y x y x y Y Z x y z Z x z Y Z y z Y Y Z for ll z with Z z > 0. Y Z x y z x for ll yz with YZ yz > 0. YZ Z z If we lrey know Z itionl informtion on Y oes not inrese the knowlege on. 6

7 Conitionl Ineenene - Exmles Seeing Fine Tye of Cr erhs Seeing Fine Tye of Cr See Aility of Tem A Aility of Tem B Aility of Tem A Aility of Tem B Outome of Tem A n B 7

8 Conitionl Ineenene Another hrteriztion of on. ineenene Proosition Y Z there exist funtions fxz n gyz suh tht YZ x y z f x z g y z for ll x y n z with Z z > 0. Corollry 2 Y there exist funtions fx n gy suh tht Y x y f x g y for ll x y. 8

9 Conitionl Ineenene Proof of Pro.. Cler from the efinition. For ny x y n z with Z z > 0 We hve Thus 9

10 Unirete Grh n Mrkov Proerty 0

11 Unirete Grh Unirete Grh G V E : unirete grh V: finite set E V V the orer is neglete. Exmle: Grh terminology V { } E { } Comlete: A sugrh S of V is omlete if ny n in S re onnete y n ege. Clique: A lique is mximl omlete suset w.r.t. inlusion. : omlete ut not lique

12 Proility n Unirete Grh Proility ssoite with n unirete grh G V E : unirete grh. V { n} K n : rnom vriles inexe y the noe set V. The roility istriution of is ssoite with G if there is nonnegtive funtion ψ C C for eh lique C in G suh tht ψ C C C:lique Nottion: for suset S of V S S An unirete grh oes not seify single roility ut efines fmily of roilities. In other wors it uts restritions y the onitionl ineenene reltions reresente y the grh. 2

13 Proility n Unirete Grh is ssoite with n unirete grh G if n only if it mits Exmle e 2 e Z ψ ψ ψ :lique C C C Z ψ ftorizes w.r.t. G. ψ C : ftor or otentil Z: normliztion onstnt

14 Mrkov Proerty Unirete grh n Mrkov roerty Sertion: G V E : unirete grh. A B S: isjoint susets of V. S sertes A from B if every th etween ny in A n in B intersets with S. B 5 A 2 4 S Theorem G V E : unirete grh. : rnom vetor with the istriution ssoite with G. If S sertes A from B then A B S Proof: next leture. 4

15 5 Mrkov Proerty Exmle { } sertes {} n {e} {} sertes {} n {} e 2 e Z ψ ψ ψ e {} 2 e e Z ψ ψ ψ ~ 2 e Z ψ ψ ψ e g f Z { } e e Z 2 ψ ψ ψ g Z ψ Use ro.. Use ro..

16 Mrkov Proerty Glol Mrkov Proerty G V E : unirete grh : rnom vetor inexe y V. stisfies glol Mrkov roerty reltive to G if A B S hols for ny trilet ABS of isjoint susets of V suh tht S sertes A from B. The revious theorem tells if the istriution of ftorizes w.r.t. G then stisfies glol Mrkov roerty reltive to G. Remrk: Both of ftorize n glol Mrkov roerty re the roerties regring reltion etween the roility n the unirete grh G. 6

17 Mrkov Proerty Hmmersley-Cliffor theorem see e.g. Luritzen. Th..9 Theorem 4 G V E : unirete grh : rnom vetor inexe y V. Assume tht the roility ensity funtion of the istriution of is stritly ositive. If stisfies glol Mrkov roerty w.r.t. G then ftorizes w.r.t. G i.e. mits the ftoriztion: ψ C C C:lique. Ftoriztion Th. Th. 4 with ositivity Glol Mrkov 7

18 Direte Ayli Grh n Mrkov Proerty 8

19 Direte Ayli Grh Direte Grh G V E : irete grh V: finite set -- noes E V V : set of eges Exmle: V { } E { } Orient the ege y Direte Ayli grh DAG Direte grh with no yles. Cyle: irete th strting n ening t the sme noe. 9

20 20 DAG n Proility Proility ssoite with DAG A DAG efines fmily of roility istriutions e e e n i i i n K : rents of noe i. { } E j i V j i Exmle: is si to e ssoite with DAG G or ftorizes w.r.t. G.

21 2 Conitionl Ineenene with DAG Three si ses 2 Note Note: re the sme for n 2.

22 Conitionl Ineenene with DAG he-to-he or v-struture Note: in re ifferent from n 2. Allergy Col Sneeze If you often sneeze ut you o not hve ol then it is more likely you hve llergy hy fever. 22

23 D-Sertion Bloke: An unirete th π is si to e loke y suset S in V if there exists noe on the th suh tht either S i n is not he-to-he in π or ii n { } e S φ. he-to-he Desenent: e i { j V irete th from i to j} Exmles S π π π S S π is loke y S π is loke y S π is NOT loke y S 2

24 D-Sertion -serte: A B S: isjoint susets of V. S -sertes A from B if every unirete th etween in A n in B is loke y S. -sertion n onitionl ineenene Theorem 5 : rnom vetor with the istriution ssoite with DAG G. A B S: isjoint susets of V. If S -sertes A from B then A B S Proof not shown in this ourse. See Luritzen 996.2&.25 24

25 D-Sertion Exmle S φ. is loke with. is loke with e is loke with e e {} is loke with. is loke with e is loke with e or e 25

26 Comrison: UDG n DAG Limittion of unirete grh DAG If ny UDG is not le to exress. 26

27 27 Comrison: UDG n DAG Limittion of DAG Unirete grh No DAG exresses these onitionl ineenene reltionshis. {} {} If every noe h the form the grh woul e yle. Thus there must e v-struture. Conitionl ineenene of the rents of the v-struture given the other two noes nnot e exresse y DAG. [Sketh of the roof.]

28 Mini Summry on UDG n DAG Unirete grh Direte yli grh DAG e e Proility ssoite with G ftorizes w.r.t. G Z ψ C C C:lique Proility ssoite with G ftorizes w.r.t. G n n i i i K ftorizes w.r.t. G ftorizes w.r.t. G is glol Mrkov reltive to G. is -glol Mrkov reltive to G. i.e. if S sertes A from B i.e. if S -sertes A from B then. then. A B S A B S 28

29 Aenix: Terminology on Grhs Unirete grh G V E Ajent: n in V re jent if Neighor: E. ne { V E}. ne DAG G V E Prents: Chilren: Anestors: Desenents: { V E}. h { V E}. n { V irete th from to }. e { V irete th from to }. n e 29

30 Ftor Grh n Mrkov Proerty 0

31 Ftor Grh i Ftor grh G V E V I F: two tyes of noes j l I: vrile noes vrile noe F: ftor noes ftor noe E: unirete eges k E I F V V. An ege exists only etween ftor noe n vriles noe. A ftor grh is in generl lle irtite grh. A irtite grh is n unirete grh G V E suh tht V V V V V φ E V. 2 2 V2

32 Proility n Ftor grh Ftor grh to reresent ftoriztion i i I : rnom vetor inexe y finite set I. The ensity of the istriution of ftorizes s f F: finite set. Z F Z: normliztion onstnt f : non-negtive funtion of suset of { n } i i I where I : { i I i E} The ftor grh G V E reresenting the ftoriztion is given y V I F E { i I F i I} 2

33 Proility n Ftor grh Exmle I {245} F {} A roility is often given y ftorize form i.e. rout of ftors with smll numer of vriles f f f Z

34 Mrkov Proerty of Ftor Grh nei: neighor of vrile noe i ne i { j I F{ i j} I }. 2 A th in ftor grh is sequene of vriles noes suh tht ny onseutive 5 4 two noes re neighors. e.g ne4 Ftoriztion glol Mrkov roerty Theorem 6 Assume the roility of ftorizes w.r.t. ftor grh G. S A B: isjoint susets of the vrile noes I. If every th etween ny in A n in B intersets with S then A B S 4

35 5 Mrkov Proerty of Ftor Grh Exmle Exmle 2 2 g f Z g f f f f Z 5 {4} Diret onfirmtion f f f Z f g f Z Z ψ ϕ Pro.

36 Comrison of Ftor Grh n other grhs Ftor grh n UDG 2 2 f 2 Z 2 f f Ftor grhs Unirete grh i ii iii UDG nnot istinguish the ftoriztion in i n ii All the vrile noes in i ii n iii hve the sme neighors n thus the sme onitionl ineenene reltionshis no onitionl ineenene. The ftor grh reresenttions of i n ii re ifferent. 6

37 Comrison of Ftor Grh n other grhs Ftor grh n DAG DAG 2 Ftor grh 2 Ineenene of n 2 nnot e reresente. 7

38 More on Mrkov Proerty 8

39 Mrkov Proerties Revisite Mrkov roerties for n unirete grh G V E : unirete grh. : rnom vetor inexe y V. V \ { } ne Lol Mrkov stisfies lol Mrkov roerty reltive to G if for ny noe V \{ } ne ne ne Pirwise Mrkov stisfies irwise Mrkov roerty reltive to G if ny non-jent ir of noes stisfies V \{ } V \{ } 9

40 Mrkov Proerties Revisite Theorem 7 Ftoriztion glol Mrkov lol Mrkov irwise Mrkov roof ftoriztion glol Mrkov : Theorem. glol Mrkov lol Mrkov : esy. lol Mrkov irwise Mrkov : nees some mth Exerise. Hmmersley-Cliffor sserts tht the irwise Mrkov roerty mens ftoriztion w.r.t. the grh uner ositivity of the ensity. Theorem 4 ssumes glol Mrkov ut the ssertion hols uner irwise Mrkov ssumtoin. Similr notions re efine for irete n ftor grhs. 40

41 Proof for Unirete Cse We show slight generliztion of Theorem. Theorem 8 Let G V E e n unirete grh. If the istriution of ftorizes s ψ C C Z C:omlete then stisfies glol Mrkov roerty reltive to G i.e. for trilet S A B suh tht S sertes A from B the onitionl ineenene A B S hols. A S B Proof A ~ Let ~ B ~ A { V \ S A π th from to π S φ} ~ ~ B V \ A S. 4

42 Proof for Unirete Cse ~ A A Oviously n sine S sertes A from B ~ B B. We n show for ny omlete sugrh C ~ ~ or hols. C S C S A C S B If there is nothing to rove. Assume C S. Suose tht the ove ssertion oes not hol then ~ ~ Let A ~ ~ C A φ n C B φ. C n B C. Beuse n re in the omlete sugrh C there is n ~ ege e onneting n. Sine A there is th π from to A without interseting S. Conneting π n e mkes th from to A without interseting S whih ontrits with the efinition of n. A ~ B ~ A S A ~ B ~ e B 42

43 Proof for Unirete Cse From this ft ψ C C ψ C C ψ Z Z C:omlete C:omlete ~ C S A D D:omlete ~ D S B D f ~ g ~ A S B S whih mens n thus ~ A ~ B S A B S. Proosition Q.E.D. 4

44 44 Converting Ftor Grh to UDG Neighorhoo struture y ftor grh mke n unirete grh. Eh ftor in A oes not orreson to lique in U ut to omlete sugrh in U. In generl ftorizes s for the onverte unirete grh U. LA f f f f Z Ftor grh G Unirete grh U :omlete C C C Z ψ

45 Proof for Ftor Grh Proof of Theorem 6 Ftoriztion Glol Mrkov for ftor grh From the ove oservtion the roof is one y Theorem 8. 45

46 Prtil Exmles Mrkov rnom fiel for imge nlysis Z i j E ex U ij i j i j Mixture moel n hien Mrkov moel Conitionl rnom fiel for sequentil t Lfferty et l. 200 Hien lel sequene Oservtion 46

47 Summry A grh reresents the onitionl ineenene reltionshis mong rnom vriles. There re mny tyes of grh to reresent roilities. Unirete grh Direte grh Ftor grh Ftoriztion of the roility istriution w.r.t. grh mens Mrkov Proerty of the istriution reltive to the grh. 47