Example: multivariate Gaussian Distribution

Transcription

1 School of omputer Science Probabilistic Graphical Models Representation of undirected GM (continued) Eric Xing Lecture 3, September 16, 2009 Reading: KF-chap4 Eric MU, Example: multivariate Gaussian istribution Gaussian distribution can be represented by a fully connected graph with pairwise (edge) potentials over continuous nodes. The overall energy has the form H ( x) = ij ( x i 1 T 1 µ ) Σ ( x µ ) = ( x µ ) Σ ( x µ ) ij j where µ is the mean and Θ is the inverse covariance (precision) matrix. lso known as Gaussian graphical model (GGM), same as oltzmann machine except x i R Eric MU,

2 Sparse precision vs. sparse covariance in GGM Σ = Σ = X 1 Σ 15 = 0 X1 X 5 X nbrs (1) or nbrs(5) 1 X 5 Σ15 = 0 Eric MU, Where is the graph structure come from? Microarray ata 1hr 2hr 3hr 4hr Eric MU,

3 Neighborhood selection It can be shown that if then Eric MU, Recall lasso Eric MU,

4 Graph Regression Lasso: Eric MU, Graph Regression Eric MU,

5 Graph Regression Eric MU, Summary Undirected graphical models capture relatedness, coupling, co-occurrence, synergism, etc. between entities Local and global independence properties identifiable via graph separation criteria efined on clique potentials Generally intractable to compute likelihood due to presence of partition function Therefore not only inference, but also likelihood-based learning is difficult in general an be used to define either joint or conditional distributions Important special cases: Gaussian graphical models Ising models Eric MU,

6 School of omputer Science Probabilistic Graphical Models ayesian & Markov Networks: unified view Receptor X 1 Receptor X 2 Eric Xing Lecture 4, September 21, 2009 Kinase Kinase Kinase E X 3 X 4 X 5 Gene G TF F X 6 X 7 Gene H X 8 Reading: Eric MU, Question: Is there a N that is a perfect map for a given MN? The "diamond" MN Eric MU,

7 Question: Is there a N that is a perfect map for a given MN? {,} {,} {,} {,} This MN does not have a perfect I-map as N! Eric MU, Question: Is there an MN that is a perfect I-map to a given N? V-structure example Eric MU,

8 Question: Is there an MN that is a perfect I-map to a given N? ( ) ( ) ( ) ( ) V-structure has no equivalent in MNs! Eric MU, Minimal I-maps Instead of attempting perfect I-maps between Ns and MNs, we can try minimal I-maps Recall: H is a minimal I-map for G if I(H) I(G) Removal of a single edge in H renders it is not an I-map Note: If H is a minimal I-map of G, H need not necessarily satisfy all the independence relationships in G Eric MU,

9 Minimal I-maps from Ns to MNs: Markov lanket Markov lanket of X in a N G: M G (X) is the unique minimal set U of nodes in G such that (X (all other nodes) U) is guaranteed to hold for any distribution that factorizes over G efn: M G (X) is the set of nodes consisting of X s parents, X s children and other parents of X s children Idea: The neighbors of X in H --- the minimal I-map of G --- should be M G (X)! Eric MU, Minimal I-maps from Ns to MNs: Moral Graphs efn (5.7.3): The moral graph M(G) of a N G is an undirected graph that contains an undirected edge between X and Y if: there is a directed edge between them in either direction X and Y are parents of the same node omment: this definition ensures M G (X) is the set of neighbors of X in M(G) Eric MU,

10 Minimal I-maps from Ns to MNs: Moral graph is the minimal I-map orollary (5.7.4): The moral graph M(G) of any N G is a minimal I- map for G Moralization turns each (X, Pa(X)) into a fully connected subset Ps associated with the network can be used as clique potentials The moral graph loses some independence information ut all independence propositions in the moral graph (careful, not including non-independence assumptions) are withheld in the N ( ) Eric MU, Minimal I-maps from Ns to MNs: Perfect I-maps Proposition (5.7.5): If the N G is "moral", then its moralized graph M(G) is a perfect I-map of G. Proof sketch: I(M(G)) I(G) (from before) The only independence relations that are potentially lost from G to M(G) are those arising from V-structures Since G has no V-structures (it is moral), no independencies are lost in M(G) Example of M(G) is a perfect I-map of G? Eric MU,

11 Soundness of d-separation Recall d-separation Let U ={X, Y, Z} be three disjoint sets of nodes in a N G. Let G + be the ancestral graph: the induced N over U ancestors(u). Then, d-sep G (X;Y Z) iff sep M(G + ) (X;Y Z) -sep G (;I L) -sep G (;I S, ) sep M(G + ) (;I L) Sep M(G + ) (;I S,) Eric MU, Soundness of d-separation Why it works: G: M(G): ( ) M(G + ): Idea: Information blocked through common children in G that are not in the conditioning variables, is simulated in M(G+) by ignoring all children. Eric MU,

12 Minimal I-maps from Ns to MNs: Summary Moral Graph M(G) is a minimal I-map of G If G is moral, then M(G) is a perfect I-map of G -sep G (X;Y Z) sep M(G + ) (X;Y Z) Next: minimal I-maps from MNs to Ns Eric MU, Minimal I-maps from MNs to Ns: ny N I-map for an MN must add triangulating edges into the graph Eric MU,

13 Minimal I-maps from MNs to Ns: chordal graphs efn (5.7.11): Let X 1 -X 2 - X k -X 1 be a loop in a graph. chord in a loop is an edge connecting X i and X j fo non-consecutive {X i, X j } n undirected graph H is chordal if any loop X 1 -X 2 - X k -X 1 for K >= 4 has a chord efn (5.7.12): directed graph G is chordal if its underlying undirected graph is chordal Eric MU, Minimal I-maps from MNs to Ns: triangulation Thm (5.7.13): Let H be an MN, and G be any N minimal I- map for H. Then G can have no immoralities. Intuitive reason: Immoralities introduce additional independencies that are not in the original MN (cf. proof for theorem in K&F) orollary (5.7.14): Let K be any minimal I-map for H. Then K is necessarily chordal! ecause any non-triangulated loop of length at least 4 in a ayesian network graph necessarily contains an immorality Process of adding edges also called triangulation Eric MU,

14 Thm (5.7.15): Let H be a non-chordal MN. Then there is no N G that is a perfect I-map for H. Proof: Minimal N I-map G for MN H is chordal It must therefore have additional directed edges not present in H Each additional edge eliminates some independence assumptions Hence proved. Eric MU, lique trees (1) Notation: Let H be a connected undirected graph. Let 1, k be the set of maximal cliques in H. Let T be a tree structured graph whose nodes are 1, k. Let i, j be two cliques in the tree connected by an edge. efine S ij = i j be the sep-set between i and j Let W <(i,j) = Variables( i ) Variables(S ij ) --- the residue set Eric MU,

15 lique trees (2) tree T is a clique tree for H if: Each node in T corresponds to a clique in H and each maximal clique in H is a node in T Each sepset S i,j separates W <(i,j) and W <(j,i) Every undirected chordal graph H has a clique tree T. Proof by induction (cf. Theorem in K&F) Example in next slide Eric MU, Example Example chordal graph and its clique tree, E, E E F EF E F,E Eric MU,

16 I-maps of MN as N: Thm (5.7.19): Let H be a chordal MN. Then there exists a N such that I(H) = I(G). Proof sketch: Since H is an MN, it has a clique tree Number the nodes consistent with clique ordering E F E E EF Eric MU, I-maps of MN as N: Thm (5.7.19): Let H be a chordal MN. Then there exists a N such that I(H) = I(G). Proof sketch (contd): For each node X i, let k be the first clique it occurs in. efine Pa(X i ) = var{ k } X i {X 1, X i-1 } F E E G and H have the same edges ll parents of each X i are in the same clique node they are connected no immoralities in G EF E Eric MU,

17 Minimal I-maps from MNs to Ns: Summary minimal I-map N of an MN is chordal Obtained by triangulating the MN If the MN is chordal, there is a perfect N I-map for the MN Obtained from the corresponding clique-tree Next: Hybrids of Ns and MNs Partially irected cyclic Graphs Eric MU, Partially irected cyclic Graphs lso called chain graphs Nodes can be disjointly partitioned into several chain components n edge within the same chain component must be undirected n edge between two nodes in different chain components must be directed hain components: {}, {}, {,,E},{F,G},{H}, {I} Eric MU,

18 Example: onditional Random Fields Y 1 Y 2 Y 3... Y T X 1 X 2 X 3... X T Y 1 Y 2 Y 3... Y T X 1 X 2 X 3... X T Y 1 Y 2 Y 5 Y 1 Y 2 Y 5 X 1 X n X 1 X n Eric MU, Example: onditional Random Fields Y 1 Y 2 Y 3... Y T iscriminative X 1 X 2 X 3... X T 1 p θ ( y x ) = exp θcfc ( x, yc ) Z ( θ, x ) c Y 1 X 1 Y 2 Y 3 X 2 X Y T X T oesn t assume that features are independent Y 1 Y 2 Y 5 When labeling X i future observations are taken into account X 1 X n Eric MU,

19 onditional Models onditional probability P(label sequence y observation sequence x) rather than joint probability P(y, x) Specify the probability of possible label sequences given an observation sequence llow arbitrary, non-independent features on the observation sequence X The probability of a transition between labels may depend on past and future observations Relax strong independence assumptions in generative models Eric MU, onditional istribution If the graph G = (V, E) of Y is a tree, the conditional distribution over the label sequence Y = y, given X = x, by the Hammersley lifford theorem of random fields is: pθ (y x) exp λ k fk( e,y e,x) + µ kgk( v,y v,x) e E,k v V,k x is a data sequence y is a label sequence v is a vertex from vertex set V = set of label random variables e is an edge from edge set E over V f k and g k are given and fixed. g k is a oolean vertex feature; f k is a oolean edge feature k is the number of features θ = ( λ, λ, L, λ ; µ, µ, L, µ ); λ andµ 1 2 n 1 2 n k k y e is the set of components of y defined by edge e y v is the set of components of y defined by vertex v are parameters to be estimated Y 1 Y 2 Y 5 X 1 X n Eric MU,

20 onditional istribution (cont d) RFs use the observation-dependent normalization Z(x) for the conditional distributions: 1 pθ (y x) = exp λ k fk( e,y e,x) + µ kgk( v,y v,x) Z(x) e E,k v V,k Z(x) is a normalization over the data sequence x Eric MU, onditional Random Fields 1 p θ ( y x ) = exp θcfc ( x, yc ) Z ( θ, x ) c llow arbitrary dependencies on input lique dependencies on labels Use approximate inference for general graphs Eric MU,

21 Summary Investigated the relationship between Ns and MNs They represent different families of independence assumptions hordal graphs can be represented in both hain networks: superset of both Ns and MNs Why we care about this: N and MN offer different semantics for designer to capture or expression (conditional) independences among variables Under certain condition N can be represented as an MN and vice versa In the future, for certain operation (i.e., inference), we will be using a single representation as the data structure for which an algorithm can operate on. This makes algorithm design, and analysis of the algorithms simpler Eric MU,