Representation of undirected GM. Kayhan Batmanghelich

Transcription

1 Representation of undirected GM Kayhan Batmanghelich

2 Review

3 Review: Directed Graphical Model Represent distribution of the form ny p(x 1,,X n = p(x i (X i i=1 Factorizes in terms of local conditional probabilities Parents of! " Each node has to maintain p(x i (X i Each variable is Conditional Independent of its non-descendants given its parents the nodes before! " that are not its parents! " Parents of! " Such an ordering is a topological ordering (i.e., parents have lower numbers than their children

4 Review: Directed Graphical Model For discrete variables, each node stores a conditional probability table (CPT

5 Review: independence properties of DAGs Defn: let I l (G be the set of local independence properties encoded by DAG G, namely: Defn: A DAG G is an I-map (independence-map of P if I l (GÍ I(P A fully connected DAG G is an I-map for any distribution, since I l (G=ÆÍ I(P for any P. 5

6 Review: I-equivalence Which graphs satisfy? I(G ={ 1? 2 3 } Defn : The skeleton of a Bayesian network graph G over V is an undirected graph over V that contains an edge {X, Y} for every edge (X, Y in G.

7 Why Undirected GM?

8 DGM is not always a good choice air or land??

9 DGM is not always a good choice What if we cannot observe h?

10 Undirected Graphical Models (UGM As in DGM, the nodes in the graph represent the variables Edges represent probabilistic interaction between neighboring variables Parametrization? In DGM we used CPD (conditional probabilities to represent distribution of a node given others For undirected graphs, we use a more symmetric parameterization that captures the affinities between related variables. Differences: Pairwise (non-causal relationships No eplicit way to generate samples 10

11 What is UGM and What are they good for?

12 Undirected graphical models (UGM X 1 X 4 X 3 X 2 X 5 Pairwise (non-causal relationships Can write down model, and score specific configurations of the graph, but no eplicit way to generate samples Contingency constrains on node configurations

13 Social networks Opinions of the students about HW0. Query: Did Tassilo like the HW0 given a few observation? Links represent correlation between classmates. Sumedha?? corr, < 0 Kayhan Mingming corr(.,. > 0 Yuanning Tassilo corr(.,. > 0 corr(.,. > 0 Yifeng Xiongtao 13

14 A Canonical Eample: understanding comple scene 14

15 Protein interaction networks Eric CMU,

16 Information retrieval topic tet image Eric CMU,

17 Undirected graphical models (UGM Defn (also called Markov Network: For a set of variables X = { 1,, n } network is defined as a product of potentials on subsets of the variables a Markov Normalizer to ensure it is a # is a probability This is called potential 0 (this does not have to be probability Maimal clique Def: A maimal clique is a clique that cannot be etended by including one more adjacent verte, meaning it is not a subset of a larger clique. 17

18 Independence

19 Remember the Markov Blanket for BN Structure: DAG Meaning: a node is conditionally independent of every other node in the network outside its Markov blanket Y 1 Y 2 Ancestor Parent X Child Children's co-parent Descendent 19

20 About Conditional Independence Global Markov Property: them if and only if C separates A from B (there is no path connecting Markov Blanket (local property is the set of nodes that renders a node! conditionally independent of all the other nodes in the graph All nodes in the graph Markov Blanket 20

21 Eample of Dependencies Pairwise: Local: Global: 21

22 Eample of Dependencies Pairwise: Local: Global: Global Local Pairwise For proof: See page 119 of the book by Koller and Friedman 22

23 UGM and DGM Probabilistic Models Graphical Models Directed Models Chordal Undirected Models Triangulation: UGM DGM Moralization: DGM UGM 23

24 Not all UGM can be represented as DGM X X In this graph, B and D are marginally independent 24

25 Not all DGM can be represented as UGM 25

26 What is this Clique?

27 Undirected graphical models (UGM Defn (also called Markov Network: For a set of variables X = { 1,, n } network is defined as a product of potentials on subsets of the variables a Markov Normalizer to ensure it is a # is a probability This is called potential 0 (this does not have to be probability Maimal clique Def: A maimal clique is a clique that cannot be etended by including one more adjacent verte, meaning it is not a subset of a larger clique. 27

28 Eamples 28

29 Interpretation of Clique Potentials X Y Z The model implies must factorize as: X? Z Y but also we can write it but also. This independence statement implies (by definition that the joint p(, y, =p(yp( yp(z y p(, yp(z y p( yp(z,y but also f 1 (, y Z 1 f 2 (y, z Z 2 29

30 Interpretation of Clique Potentials X Y Z The model implies must factorize as: X? Z Y. This independence statement implies (by definition that the joint Take-home message p(, about y, potentials: =p(yp( yp(z y Those are not necessarily marginals or conditionals. but The also positive we can write clique it potentials p(, can only yp(z y be thought of as general but "compatibility", also "goodness" or p( yp(z,y "happiness" functions over their variables, but not as probability distributions. but also f 1 (, y f 2 (y, z Z 1 Z 2 30

31 Eample UGM using ma cliques A D B A,B,D B,C,D C y ( 124 c y c ( P' ( 1, 2, 3, 4 = y c ( 124 y c ( 234 Z Z =, 1 å y c ( 124 y c ( 2, 3, 4 For discrete nodes, we can represent P(X 1:4 as two 3D tables instead of one 4D table 234 Eric CMU,

32 Eample UGM using subcliques A A,D D B A,B B,D C,D Z = å Õ 1, 2, 3, 4 ij y ( ij ij C P"( 1, 2, 3, 4 1 = Õy ij ( Z ij 1 = y Z 12 ( 12 ij y B,C 14 ( 14 y 23 ( 23 y 24 ( 24 y 34 ( 34 We can represent P(X 1:4 as 5 2D tables instead of one 4D table Pair MRFs, a popular and simple special case Are two graphs equivalent ( I(P 0 and I(P 00? Eric CMU,

33 Eample UGM canonical representation A C D B ( ( ( ( ( ( ( ( ( ( (,,, ( Z P c c y y y y y y y y y y y = å = c c Z,,, ( ( ( ( ( ( ( ( ( ( ( y y y y y y y y y y y l Most general, subsume P' and P" as special cases Eric CMU,

34 Hammersley-Clifford Theorem If arbitrary potentials are utilized in the following product formula for probabilities, 1 P (,, = y ( 1! Z = 1 n Z å Õ,!, n cîc Õ cîc y ( c c c c then the family of probability distributions obtained is eactly that set which respects the qualitative specification (the conditional independence relations described earlier Thm : Let P be a positive distribution over V, and H a Markov network graph over V. If H is an I-map for P, then P is a Gibbs distribution over H. Eric CMU,

35 Factor Graphs

36 Factor Graph Random variables 1 f( 1, f( 1, 3 4 f(2, 4 Factors A factor graph is a graphical model representation that unifies directed and undirected models It is an undirected bipartite graph with two kinds of nodes. Round nodes represent variables, Square nodes represent factors and there is an edge from each variable to every factor that mentions it. Represents the distribution more uniquely than a graphical model

37 Factor Graph for UGM

38 Factor Graph for DGM One factor per CPD (conditional distribution and connect the factor to all the variables that use the CPD

39 Practical Eamples

40 Eponential Form p( 1,, n = 1 CY c(x c So-called Potentials > 0 Z p( 1,, n = 1 c=1 CY ep ( c(x c Z Remember the Gibbs distribution: c=1 Free Energy of the system (log of prob: H( 1,, n = X c A powerful parametrization (log-linear model: Energy of the clique, can be positive/negative c(x c H( 1,, n ; = X Param c f c (X c T c Feature function 40

41 Eample: Boltzmann machines A fully connected graph with pairwise (edge potentials on binary-valued nodes (for! " { 1, +1} or! " {0,1} is called a Boltzmann machine 2 1 p( 1, 2, 3, 4 ; ; = Z(, ep 4 X ij i j + X ij i 3 i i Hence the overall energy function has a quadratic form. H(;,µ=( µ T ( µ 41

42 Ising models Nodes are arranged in a regular topology (often a regular packing grid and connected only to their geometric neighbors. 1 ì p( X = epí åqij X i X j + åqi0 X Z îi, jîni i i ü ý þ Same as sparse Boltzmann machine, where q ij ¹0 iff i,j are neighbors. e.g., nodes are piels, potential function encourages nearby piels to have similar intensities. Potts model: multi-state Ising model. Eric CMU,

43 Restricted Boltzmann Machines (RBM Observed can piels, signal in speech, word in a document Unobserved has a notion of summary of data One can use it as building block for more complicated models hidden units (h visible units ( 0 1 p(, h; =ep@ X i i i (+ X j j j (h j + X i,j i,j ( i,h j A( A 43

44 Properties of RBM Factors are marginally dependent. Factors are conditionally independent given observations on the visible nodes. p(h 1,,h M = Y m p(h m Iterative Gibbs sampling to generate pairs of (,h. Learning with contrastive divergence h ~ p( h ~ p( h Eric CMU,

45 Conditional Random Fields Y 1 A X 1 Y 1 Y 2 Y 3 A A X 2 X 3 Y 2 Y Y T A X T Y T For eample: part of speech labeling We are interested in Discriminative (not joint: 1 ì ü p = íå q ( y ep qc fc (, yc ý Z( q, î c þ A X 1 A A X 2 X 3... A X T Y 1 Y 2 Y 5 X 1 X n Eric CMU,

46 Summary Undirected graphical models capture relatedness, coupling, co-occurrence, synergism, etc. between entities Local and global independence properties identifiable via graph separation criteria Defined on clique potentials Can be used to define either joint or conditional distributions Generally intractable to compute likelihood due to presence of partition function Therefore not only inference, but also likelihood-based learning is difficult in general Important special cases: Ising models RBM CRF Eric CMU,