Graphical Models - Part II - PDF Free Download

Graphical Models - Part II Bishop PRML Ch. 8 Alireza Ghane

Outline Probabilistic Models Bayesian Networks Markov Random Fields Inference Graphical Models Alireza Ghane / Greg Mori 1

Outline Probabilistic Models Bayesian Networks Markov Random Fields Inference Graphical Models Alireza Ghane / Greg Mori 2

Joint Distribution In general, to answer a query on random variables Q = Q 1,..., Q N given evidence E = e, E = E 1,..., E M, e = e 1,..., e M : p(q E = e) = = p(q, E = e) p(e = e) h p(q, E = e, H = h) p(q = q, E = e, H = h) q,h Graphical Models Alireza Ghane / Greg Mori 3

Problems The joint distribution is large e. g. with K boolean random variables, 2 K entries Inference is slow, previous summations take O(2 K ) time Learning is difficult, data for 2 K parameters Analogous problems for continuous random variables Graphical Models Alireza Ghane / Greg Mori 4

Reminder - Independence Cavity Toothache Catch Weather decomposes into Cavity Toothache Catch Weather A and B are independent iff p(a B) = p(a) or p(b A) = p(b) or p(a, B) = p(a)p(b) p(t oothache, Catch, Cavity, W eather) = p(t oothache, Catch, Cavity)p(W eather) 32 entries reduced to 12 (W eather takes one of 4 values) Absolute independence powerful but rare Dentistry is a large field with hundreds of variables, none of which are independent. What to do? Graphical Models Alireza Ghane / Greg Mori 5

Graphical Models Graphical Models provide a visual depiction of probabilistic model Conditional indepence assumptions can be seen in graph Inference and learning algorithms can be expressed in terms of graph operations We will look at 2 types of graph (can be combined) Directed graphs: Bayesian networks Undirected graphs: Markov Random Fields Factor graphs (won t cover) Graphical Models Alireza Ghane / Greg Mori 6

Outline Probabilistic Models Bayesian Networks Markov Random Fields Inference Graphical Models Alireza Ghane / Greg Mori 7

Bayesian Networks A simple, graphical notation for conditional independence assertions and hence for compact specification of full joint distributions Syntax: a set of nodes, one per variable a directed, acyclic graph (link directly influences ) a conditional distribution for each node given its parents: p(x i pa(x i )) In the simplest case, conditional distribution represented as a conditional probability table (CPT) giving the distribution over X i for each combination of parent values Graphical Models Alireza Ghane / Greg Mori 8

Example I m at work, neighbor John calls to say my alarm is ringing, but neighbor Mary doesn t call. Sometimes it s set off by minor earthquakes. Is there a burglar? Variables: Burglar, Earthquake, Alarm, JohnCalls, MaryCalls Network topology reflects causal knowledge: A burglar can set the alarm off An earthquake can set the alarm off The alarm can cause Mary to call The alarm can cause John to call Graphical Models Alireza Ghane / Greg Mori 9

Example contd. Burglary P(B).001 Earthquake P(E).002 B T T F F E T F T F P(A B,E).95.94.29.001 Alarm JohnCalls A T F P(J A).90.05 MaryCalls A T F P(M A).70.01 Graphical Models Alireza Ghane / Greg Mori 10

Global Semantics Global semantics defines the full joint distribution as the product of the local conditional distributions: B A E P (x 1,..., x n ) = n P (x i pa(x i )) i=1 J M e.g., P (j m a b e) = P (j a)p (m a)p (a b, e)p ( b)p ( e) = 0.9 0.7 0.001 0.999 0.998 0.00063 Graphical Models Alireza Ghane / Greg Mori 11

Constructing Bayesian Networks Need a method such that a series of locally testable assertions of conditional independence guarantees the required global semantics 1. Choose an ordering of variables X 1,..., X n 2. For i = 1 to n add X i to the network select parents from X 1,..., X i 1 such that p(x i pa(x i )) = p(x i X 1,..., X i 1 ) This choice of parents guarantees the global semantics: p(x 1,..., X n ) = = n p(x i X 1,..., X i 1 ) (chain rule) i=1 n p(x i pa(x i )) i=1 (by construction) Graphical Models Alireza Ghane / Greg Mori 12

Example Suppose we choose the ordering M, J, A, B, E MaryCalls JohnCalls P (J M) = P (J)? Graphical Models Alireza Ghane / Greg Mori 14

Example Suppose we choose the ordering M, J, A, B, E MaryCalls JohnCalls Alarm P (J M) = P (J)? No P (A J, M) = P (A J)? P (A J, M) = P (A)? Graphical Models Alireza Ghane / Greg Mori 15

Example Suppose we choose the ordering M, J, A, B, E MaryCalls JohnCalls Alarm Burglary P (J M) = P (J)? No P (A J, M) = P (A J)? P (A J, M) = P (A)? P (B A, J, M) = P (B A)? P (B A, J, M) = P (B)? No Graphical Models Alireza Ghane / Greg Mori 16

Example Suppose we choose the ordering M, J, A, B, E MaryCalls JohnCalls Alarm Burglary Earthquake P (J M) = P (J)? No P (A J, M) = P (A J)? P (A J, M) = P (A)? No P (B A, J, M) = P (B A)? Yes P (B A, J, M) = P (B)? No P (E B, A, J, M) = P (E A)? Graphical P (E B, Models A, J, M) = P (E A, B)? Alireza Ghane / Greg Mori 17

Example Suppose we choose the ordering M, J, A, B, E MaryCalls JohnCalls Alarm Burglary Earthquake P (J M) = P (J)? No P (A J, M) = P (A J)? P (A J, M) = P (A)? P (B A, J, M) = P (B A)? Yes P (B A, J, M) = P (B)? No P (E B, A, J, M) = P (E A)? No P (E B, A, J, M) = P (E A, B)? Yes Graphical Models Alireza Ghane / Greg Mori 18 No

Example contd. MaryCalls JohnCalls Alarm Burglary Earthquake Deciding conditional independence is hard in noncausal directions (Causal models and conditional independence seem hardwired for humans!) Assessing conditional probabilities is hard in noncausal directions Network is less compact: 1 + 2 + 4 + 2 + 4 = 13 numbers needed Graphical Models Alireza Ghane / Greg Mori 19

Example - Car Insurance Age GoodStudent RiskAversion SeniorTrain SocioEcon Mileage VehicleYear ExtraCar DrivingSkill MakeModel DrivingHist Antilock DrivQuality Airbag CarValue HomeBase AntiTheft Ruggedness Accident OwnDamage Theft Cushioning OtherCost OwnCost MedicalCost LiabilityCost PropertyCost http://aispace.org/bayes Graphical Models Alireza Ghane / Greg Mori 20

Example - Polynomial Regression w t 1 t N Bayesian polynomial regression model Observations t = (t 1,..., t N ) Vector of coefficients w Inputs x and noise variance σ 2 were assumed fixed, not stochastic and hence not in model Joint distribution: p(t, w) = p(w) N p(t n w) n=1 Graphical Models Alireza Ghane / Greg Mori 21

Plates w w t 1 t N = t n N A shorthand for writing repeated nodes such as the t n uses plates Graphical Models Alireza Ghane / Greg Mori 22

Deterministic Model Parameters x n α σ 2 t n N w Can also include deterministic parameters (not stochastic) as small nodes Bayesian polynomial regression model: p(t, w x, α, σ 2 ) = p(w α) N p(t n w, x n, σ 2 ) n=1 Graphical Models Alireza Ghane / Greg Mori 23

Observations x n α σ 2 t n N w In polynomial regression, we assumed we had a training set of N pairs (x n, t n ) Convention is to use shaded nodes for observed random variables Graphical Models Alireza Ghane / Greg Mori 24

Predictions x n α w t n N σ 2 ˆt ˆx Suppose we wished to predict the value ˆt for a new input ˆx The Bayesian network used for this inference task would be this one Graphical Models Alireza Ghane / Greg Mori 25

Specifying Distributions - Discrete Variables Earlier we saw the use of conditional probability tables (CPT) for specifying a distribution over discrete random variables with discrete-valued parents For a variable with no parents, with K possible states: B T T F F E T F T F Burglary P(A B,E).95.94.29.001 JohnCalls P(B).001 Alarm A T F P(J A).90.05 Earthquake MaryCalls P(E).002 A P(M A) T.70 F.01 p(x µ) = K k=1 µ x k k e.g. p(b) = 0.001 B1 0.999 B2, 1-of-K representation Graphical Models Alireza Ghane / Greg Mori 26

Specifying Distributions - Discrete Variables cont. With two variables x 1, x 2 can have two cases x 1 x 2 x 1 x 2 Dependent Independent p(x 1, x 2 µ) = p(x 1 µ)p(x 2 x 1, µ) ( K ) K K = k=1 µ x 1k k1 k=1 j=1 K 2 1 free parameters in µ µ x 1kx 2j kj2 p(x 1, x 2 µ) = p(x 1 µ)p(x 2 µ) ( K ) ( K ) = k=1 µ x 1k k1 k=1 µ x 2k k2 2(K 1) free parameters in µ Graphical Models Alireza Ghane / Greg Mori 27

Chains of Nodes x 1 x 2 x M With M nodes, could form a chain as shown above Number of parameters is: (K 1) +(M 1) K(K 1) }{{}}{{} x 1 others Compare to: K M 1 for fully connected graph M(K 1) for graph with no edges (all independent) Graphical Models Alireza Ghane / Greg Mori 28

Sharing Parameters µ 1 µ 2 µ M x 1 x 2 x M µ 1 µ x 1 x 2 x M Another way to reduce number of parameters is sharing parameters (a. k. a. tying of parameters) Lower graph reuses same µ for nodes 2-M µ is a random variable in this network, could also be deterministic (K 1) + K(K 1) parameters Graphical Models Alireza Ghane / Greg Mori 29

Specifying Distributions - Continuous Variables P(c h, subsidy) 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0 0 2 4 6 8 Cost c 10 12 0 2 4 6 8 1012 Harvest h One common type of conditional distribution for continuous variables is the linear-gaussian p(x i pa i ) = N x i ; w ij x j + b i, v i j pa i Graphical Models Alireza Ghane / Greg Mori 30

Linear Gaussian Interesting fact: if all nodes in a Bayesian Network are linear Gaussian, joint distribution is a multivariate Gaussian p(x i pa i ) = N x i ; w ij x j + b i, v i j pa i N p(x 1,..., x N ) = N x i ; w ij x j + b i, v i j pa i i=1 Each factor looks like exp((x i (w T i x pa i ) 2 ), this product will be another quadratic form With no links in graph, end up with diagonal covariance matrix With fully connected graph, end up with full covariance matrix Graphical Models Alireza Ghane / Greg Mori 32

Conditional Independence in Bayesian Networks Recall again that a and b are conditionally independent given c (a b c) if p(a b, c) = p(a c) or equivalently p(a, b c) = p(a c)p(b c) Before we stated that links in a graph are directly influences We now develop a correct notion of links, in terms of the conditional independences they represent This will be useful for general-purpose inference methods Graphical Models Alireza Ghane / Greg Mori 33

A Tale of Three Graphs - Part 1 c a b The graph above means p(a, b, c) = p(a c)p(b c)p(c) p(a, b) = p(a c)p(b c)p(c) c p(a)p(b) in general So a and b not independent Graphical Models Alireza Ghane / Greg Mori 34

A Tale of Three Graphs - Part 1 c a b However, conditioned on c p(a, b c) = p(a, b, c) p(c) = p(a c)p(b c)p(c) p(c) = p(a c)p(b c) So a b c Graphical Models Alireza Ghane / Greg Mori 35

A Tale of Three Graphs - Part 1 c c a b a b Note the path from a to b in the graph When c is not observed, path is open, a and b not independent When c is observed, path is blocked, a and b independent In this case c is tail-to-tail with respect to this path Graphical Models Alireza Ghane / Greg Mori 36

A Tale of Three Graphs - Part 2 a c b The graph above means p(a, b, c) = p(a)p(b c)p(c a) Again a and b not independent Graphical Models Alireza Ghane / Greg Mori 37

A Tale of Three Graphs - Part 2 a c b However, conditioned on c So a b c p(a, b c) = p(a, b, c) p(c) = p(a)p(b c) p(c) = p(a c)p(b c) = p(a)p(b c) p(c a) p(c) p(a c)p(c) } p(a) {{ } Bayes Rule Graphical Models Alireza Ghane / Greg Mori 38

A Tale of Three Graphs - Part 2 a c b a c b As before, the path from a to b in the graph When c is not observed, path is open, a and b not independent When c is observed, path is blocked, a and b independent In this case c is head-to-tail with respect to this path Graphical Models Alireza Ghane / Greg Mori 39

A Tale of Three Graphs - Part 3 a b c The graph above means p(a, b, c) = p(a)p(b)p(c a, b) p(a, b) = p(a)p(b)p(c a, b) c = p(a)p(b) This time a and b are independent Graphical Models Alireza Ghane / Greg Mori 40

A Tale of Three Graphs - Part 3 a b c However, conditioned on c p(a, b c) = p(a, b, c) p(a)p(b)p(c a, b) = p(c) p(c) p(a c)p(b c) in general So a b c Graphical Models Alireza Ghane / Greg Mori 41

A Tale of Three Graphs - Part 3 a b a b c Frustratingly, the behaviour here is different When c is not observed, path is blocked, a and b independent When c is observed, path is unblocked, a and b not independent In this case c is head-to-head with respect to this path Situation is in fact more complex, path is unblocked if any descendent of c is observed c Graphical Models Alireza Ghane / Greg Mori 42

Part 3 - Intuition B F B F B F G G G Binary random variables B (battery charged), F (fuel tank full), G (fuel gauge reads full) B and F independent But if we observe G = 0 (false) things change e.g. p(f = 0 G = 0, B = 0) could be less than p(f = 0 G = 0), as B = 0 explains away the fact that the gauge reads empty Recall that p(f G, B) = p(f G) is another F B G Graphical Models Alireza Ghane / Greg Mori 43

D-separation A general statement of conditional independence For sets of nodes A, B, C, check all paths from A to B in graph If all paths are blocked, then A B C Path is blocked if: Arrows meet head-to-tail or tail-to-tail at a node in C Arrows meet head-to-head at a node, and neither node nor any descendent is in C Graphical Models Alireza Ghane / Greg Mori 44

Naive Bayes z x 1 x D Commonly used naive Bayes classification model Class label z, features x 1,..., x D Model assumes features independent given class label Tail-to-tail at z, blocks path between features Graphical Models Alireza Ghane / Greg Mori 45

Markov Blanket x i What is the minimal set of nodes which makes a node x i conditionally independent from the rest of the graph? x i s parents, children, and children s parents (co-parents) Define this set MB, and consider: p(x i x {j i} ) = = p(x 1,..., x D ) p(x1,..., x D )dx i k p(x k pa k ) k p(x k pa k )dx i All factors other than those for which x i is x k or in pa k cancel Graphical Models Alireza Ghane / Greg Mori 46

Learning Parameters When all random variables are observed in training data, relatively straight-forward Distribution factors, all factors observed e.g. Maximum likelihood used to set parameters of each distribution p(x i pa i ) separately When some random variables not observed, it s tricky This is a common case Expectation-maximization(EM) is a method for this Graphical Models Alireza Ghane / Greg Mori 47

Outline Probabilistic Models Bayesian Networks Markov Random Fields Inference Graphical Models Alireza Ghane / Greg Mori 48

Conditional Independence in Graphs a b a b c Recall that for Bayesian Networks, conditional independence was a bit complicated d-separation with head-to-head links We would like to construct a graphical representation such that conditional independence is straight-forward path checking c Graphical Models Alireza Ghane / Greg Mori 49

Markov Random Fields A C B Markov random fields (MRFs) contain one node per variable Undirected graph over these nodes Conditional independence will be given by simple separation, blockage by observing a node on a path e.g. in above graph, A B C Graphical Models Alireza Ghane / Greg Mori 50

Markov Blanket Markov With this simple check for conditional independence, Markov blanket is also simple Recall Markov blanket MB of x i is set of nodes such that x i conditionally independent from rest of graph given MB Markov blanket is neighbours Graphical Models Alireza Ghane / Greg Mori 51

MRF Factorization Remember that graphical models define a factorization of the joint distribution What should be the factorization so that we end up with the simple conditional independence check? For x i and x j not connected by an edge in graph: x i x j x \{i,j} So there should not be any factor ψ(x i, x j ) in the factorized form of the joint Graphical Models Alireza Ghane / Greg Mori 52

Cliques A clique in a graph is a subset of nodes such that there is a link between every pair of nodes in the subset A maximal clique is a clique for which one cannot add another node and have the set remain a clique x 1 x 2 x 3 x 4 Graphical Models Alireza Ghane / Greg Mori 53

MRF Joint Distribution Note that nodes in a clique cannot be made conditionally independent from each other So defining factors ψ( ) on nodes in a clique is safe The joint distribution for a Markov random field is: p(x 1,..., x K ) = 1 ψ C (x C ) Z where x C is the set of nodes in clique C, and the product runs over all maximal cliques Each ψ C (x C ) 0 Z is a normalization constant C Graphical Models Alireza Ghane / Greg Mori 54

MRF Joint Distribution Example The joint distribution for a Markov random field is: p(x 1,..., x 4 ) = 1 ψ C (x C ) Z C = 1 Z ψ 123(x 1, x 2, x 3 )ψ 234 (x 2, x 3, x 4 ) Note that maximal cliques subsume smaller ones: ψ 123 (x 1, x 2, x 3 ) could include ψ 12 (x 1, x 2 ), though sometimes smaller cliques are explicitly used for clarity x 1 x 2 x 3 x 4 Graphical Models Alireza Ghane / Greg Mori 55

MRF Joint - Terminology The joint distribution for a Markov random field is: p(x 1,..., x K ) = 1 ψ C (x C ) Z Each ψ C (x C ) is called a potential function Z, the normalization constant, is called the partition function: Z = ψ C (x C ) x C C Z is very costly to compute, since it is a sum/integral over all possible states for all variables in x Don t always need to evaluate it though, will cancel for computing conditional probabilities Graphical Models Alireza Ghane / Greg Mori 56

Hammersley-Clifford The definition of the joint: p(x 1,..., x K ) = 1 ψ C (x C ) Z Note that we started with particular conditional independences We then formulated the factorization based on clique potentials This formulation resulted in the right conditional independences The converse is true as well, any distribution with the conditional independences given by the undirected graph can be represented using a product of clique potentials This is the Hammersley-Clifford theorem C Graphical Models Alireza Ghane / Greg Mori 57

Energy Functions Often use exponential, which is non-negative, to define potential functions: ψ C (x C ) = exp{ E C (x C )} Minus sign by convention E C (x C ) is called an energy function From physics, low energy = high probability This exponential representation is known as the Boltzmann distribution Graphical Models Alireza Ghane / Greg Mori 58

Energy Functions - Intuition Joint distribution nicely rearranges as p(x 1,..., x K ) = 1 ψ C (x C ) Z C = 1 Z exp{ C E C (x C )} Intuition about potential functions: ψ C are describing good (low energy) sets of states for adjacent nodes An example of this is next Graphical Models Alireza Ghane / Greg Mori 59

Image Denoising Consider the problem of trying to correct (denoise) an image that has been corrupted Assume image is binary Observed (noisy) pixel values y i { 1, +1} Unobserved true pixel values x i { 1, +1} Graphical Models Alireza Ghane / Greg Mori 60

Image Denoising - Graphical Model y i x i Graphical Models Alireza Ghane / Greg Mori 61

Image Denoising - Graphical Model y i x i Cliques containing each true pixel value x i { 1, +1} and observed value y i { 1, +1} Observed pixel value is usually same as true pixel value Energy function ηx i y i, η > 0, lower energy (better) if x i = y i Cliques containing adjacent true pixel values x i, x j Nearby pixel values are usually the same Energy function βx i x j, β > 0, lower energy (better) if x i = x j Graphical Models Alireza Ghane / Greg Mori 63

Image Denoising - Graphical Model yi xi Complete energy function: E(x, y) = β {i,j} x i x j η i x i y i Joint distribution: p(x, y) = 1 exp{ E(x, y)} Z Or, as potential functions ψ n (x i, x j ) = exp(βx i x j ), ψ p (x i, y i ) = exp(ηx i y i ): p(x, y) = 1 ψ n (x i, x j ) ψ p (x i, y i ) Z i,j i Graphical Models Alireza Ghane / Greg Mori 64

Image Denoising - Inference The denoising query is arg max x p(x y) Two approaches: Iterated conditional modes (ICM): hill climbing in x, one variable x i at a time Simple to compute, Markov blanket is just observation plus neighbouring pixels Graph cuts: formulate as max-flow/min-cut problem, exact inference (for this graph) Graphical Models Alireza Ghane / Greg Mori 65

Converting Directed Graphs into Undirected Graphs x 1 x 2 x N 1 x N Consider a simple directed chain graph: p(x) = p(x 1 )p(x 2 x 1 )p(x 3 x 2 )... p(x N x N 1 ) Can convert to undirected graph Graphical Models Alireza Ghane / Greg Mori 66

Converting Directed Graphs into Undirected Graphs x 1 x 2 x N 1 x N Consider a simple directed chain graph: p(x) = p(x 1 )p(x 2 x 1 )p(x 3 x 2 )... p(x N x N 1 ) Can convert to undirected graph p(x) = 1 Z ψ 1,2(x 1, x 2 )ψ 2,3 (x 2, x 3 )... ψ N 1,N (x N 1, x N ) where ψ 1,2 = p(x 1 )p(x 2 x 1 ), all other ψ k 1,k = p(x k x k 1 ), Z = 1 Graphical Models Alireza Ghane / Greg Mori 67

Converting Directed Graphs into Undirected Graphs The chain was straight-forward because for each conditional p(x i pa i ), nodes x i pa i were contained in one clique Hence we could define that clique potential to include that conditional For a general undirected graph we can force this to occur by marrying the parents Add links between all parents in pa i This process known as moralization, creating a moral graph Graphical Models Alireza Ghane / Greg Mori 68

Strong Morals x 1 x 3 x 2 x 1 x 3 x 2 x 4 x 4 Start with directed graph on left Add undirected edges between all parents of each node Remove directionality from original edges Graphical Models Alireza Ghane / Greg Mori 69

Constructing Potential Functions x 1 x 3 x 2 x 1 x 3 x 2 x 4 x 4 Initialize all potential functions to be 1 With moral graph, for each p(x i pa i ), there is at least one clique which contains all of x i pa i Multiply p(x i pa i ) into potential function for one of these cliques Z = 1 again since: p(x) = C ψ C (x C ) = i p(x i pa i ) which is already normalized Graphical Models Alireza Ghane / Greg Mori 70

Equivalence Between Graph Types Note that the moralized undirected graph loses some of the conditional independence statements of the directed graph Further, there are certain conditional independence assumptions which can be represented by directed graphs which cannot be represented by indirected graphs, and vice versa Graphical Models Alireza Ghane / Greg Mori 71

Equivalence Between Graph Types A B C Note that the moralized undirected graph loses some of the conditional independence statements of the directed graph Further, there are certain conditional independence assumptions which can be represented by directed graphs which cannot be represented by indirected graphs, and vice versa Graphical Models Alireza Ghane / Greg Mori 72

Equivalence Between Graph Types C A B A B C D Note that the moralized undirected graph loses some of the conditional independence statements of the directed graph Further, there are certain conditional independence assumptions which can be represented by directed graphs which cannot be represented by indirected graphs, and vice versa Directed graph: A B, A B C, cannot be represented using undirected graph Graphical Models Alireza Ghane / Greg Mori 74

Outline Probabilistic Models Bayesian Networks Markov Random Fields Inference Graphical Models Alireza Ghane / Greg Mori 76

Inference Inference is the process of answering queries such as p(x n x e = e) We will focus on computing marginal posterior distributions over single variables x n using p(x n x e = e) p(x n, x e = e) The proportionality constant can be obtained by enforcing x n p(x n x e = e) = 1 Graphical Models Alireza Ghane / Greg Mori 77

Inference on a Chain Consider a simple undirected chain For inference, we want to compute p(x n, x e = e) First, we ll show how to compute p(x n ) p(x n, x e = e) will be a simple modification of this Graphical Models Alireza Ghane / Greg Mori 78

Inference on a Chain The naive method of computing the marginal p(x n ) is to write down the factored form of the joint, and marginalize (sum out) all other variables: p(x n ) =...... p(x) x 1 x n 1 x n+1 x N =... 1... ψ C (x C ) Z x 1 x n 1 x n+1 x N This would be slow O(K N ) work if each variable could take K values C Graphical Models Alireza Ghane / Greg Mori 79

Inference on a Chain However, due to the factorization terms in this summation can be rearranged nicely This will lead to efficient algorithms Graphical Models Alireza Ghane / Greg Mori 80

Simple Algebra This efficiency comes from a very simple distributivity Or more complicated version n i=1 j=1 ab + ac = a(b + c) n a i b j = a 1 b 1 + a 1 b 2 +... + a n b n = (a 1 +... + a n )(b 1 +... + b n ) Much faster to do right hand side (2(n 1) additions, 1 multiplication) than left hand side (n 2 multiplications, n 2 1 additions) Graphical Models Alireza Ghane / Greg Mori 81

A Simple Chain First consider a chain with 3 nodes, and computing p(x 3 ): p(x 3 ) = ψ 12 (x 1, x 2 )ψ 23 (x 2, x 3 ) x 1 x 2 = [ ] ψ 23 (x 2, x 3 ) ψ 12 (x 1, x 2 ) x 2 x 1 Graphical Models Alireza Ghane / Greg Mori 82

Performing the sums p(x 3 ) = x 2 ψ 23 (x 2, x 3 ) [ x 1 ψ 12 (x 1, x 2 ) ] For example, if x i are binary: [ ] a b ψ 12 (x 1, x 2 ) = x 1 c d }{{} x 2 [ ] s t ψ 23 (x 2, x 3 ) = x 2 u v }{{} x 3 Graphical Models Alireza Ghane / Greg Mori 83

Performing the sums p(x 3 ) = x 2 ψ 23 (x 2, x 3 ) For example, if x i are binary: [ ] a b ψ 12 (x 1, x 2 ) = x 1 c d }{{} x 2 [ x 1 ψ 12 (x 1, x 2 ) ] [ s ψ 23 (x 2, x 3 ) = x 2 u t ] } v {{ } x 3 x 1 ψ 12 (x 1, x 2 ) = [ a + c b + d ] }{{} x 2 µ 12(x 2 ) [ ] s(a + c) t(a + c) ψ 23 (x 2, x 3 ) µ 12 (x 2 ) = x 2 u(b + d) v(b + d) }{{} x 3 Graphical Models Alireza Ghane / Greg Mori 85

Complexity of Inference There were two types of operations Summation x 1 ψ 12 (x 1, x 2 ) K K numbers in ψ 12, takes O(K 2 ) time Multiplication ψ 23 (x 2, x 3 ) µ 12 (x 2 ) Again O(K 2 ) work For a chain of length N, we repeat these operations N 1 times each O(NK 2 ) work, versus O(N K ) for naive evaluation Graphical Models Alireza Ghane / Greg Mori 87

More complicated chain Now consider a 5 node chain, again asking for p(x 3 ) p(x 3 ) = x 1 x 2 x 4 x 5 ψ 12 (x 1, x 2 )ψ 23 (x 2, x 3 )ψ 34 (x 3, x 4 )ψ 45 (x 4, x 5 ) Graphical Models Alireza Ghane / Greg Mori 88

More complicated chain Now consider a 5 node chain, again asking for p(x 3 ) p(x 3 ) = x 1 ψ 12 (x 1, x 2 )ψ 23 (x 2, x 3 )ψ 34 (x 3, x 4 )ψ 45 (x 4, x 5 ) x 2 x 4 x 5 = ψ 12 (x 1, x 2 )ψ 23 (x 2, x 3 ) ψ 34 (x 3, x 4 )ψ 45 (x 4, x 5 ) x 2 x 1 x 4 x 5 [ ] [ = ψ 12 (x 1, x 2 )ψ 23 (x 2, x 3 ) ψ 34 (x 3, x 4 )ψ 45 (x 4, x 5 x 1 x 5 x 2 x 4 Graphical Models Alireza Ghane / Greg Mori 90

Message Passing µ α (x n 1 ) µ α (x n ) µ β (x n ) µ β (x n+1 ) x 1 x n 1 x n x n+1 x N The factors can be thought of as messages being passed between nodes in the graph µ 12 (x 2 ) x 1 ψ 12 (x 1, x 2 ) is a message passed from node x 1 to node x 2 containing all information in node x 1 In general, µ k 1,k (x k ) = x k 1 ψ k 1,k (x k 1, x k )µ k 2,k 1 (x k 1 ) Possible to do so because of conditional independence Graphical Models Alireza Ghane / Greg Mori 92

Computing All Marginals µ α (x n 1 ) µ α (x n ) µ β (x n ) µ β (x n+1 ) x 1 x n 1 x n x n+1 x N Computing one marginal p(x n ) takes O(NK 2 ) time Naively running same algorithms for all nodes in a chain would take O(N 2 K 2 ) time But this isn t necessary, same messages can be reused at all nodes in the chain Pass all messages from one end of the chain to the other, pass all messages in the other direction too Can compute marginal at any node by multiplying the two messages delivered to the node 2(N 1)K 2 work, twice as much as for just one node Graphical Models Alireza Ghane / Greg Mori 93

Including Evidence If a node x k 1 = e is observed, simply clamp to observed value rather than summing: µ k 1,k (x k ) = x k 1 ψ k 1,k (x k 1, x k )µ k 2,k 1 (x k 1 ) becomes µ k 1,k (x k ) = ψ k 1,k (x k 1 = e, x k )µ k 2,k 1 (x k 1 = e) Graphical Models Alireza Ghane / Greg Mori 94

Trees The algorithm for a tree-structured graph is very similar to that for chains Formulation in PRML uses factor graphs, we ll just give the intuition here Consider calcuating the marginal p(x n ) for the center node of the graph at right Treat x n as root of tree, pass messages from leaf nodes up to root Graphical Models Alireza Ghane / Greg Mori 95

Trees Message passing similar to that in chains, but possibly multiple messages reaching a node With multiple messages, multiply them together As before, sum out variables µ k 1,k (x k ) = x k 1 ψ k 1,k (x k 1, x k )µ k 2,k 1 (x k 1 ) Known as sum-product algorithm Complexity still O(NK 2 ) Graphical Models Alireza Ghane / Greg Mori 96

Most Likely Configuration A similar algorithm exists for finding arg max x 1,...,x N p(x 1,..., x N ) Replace summation operations with maximize operations Maximum of products at each node Known as max-sum, since often take log-probability to avoid underflow errors Graphical Models Alireza Ghane / Greg Mori 97

General Graphs Junction tree algorithm is an exact inference method for arbitrary graphs A particular tree structure defined over cliques of variables Inference ends up being exponential in maximum clique size Therefore slow in many cases Approximate inference techniques Loopy belief propagation: run message passing scheme (sum-product) for a while Sometimes works Not guaranteed to converge Variational methods: approximate desired distribution using analytically simple forms, find parameters to make these forms similar to actual desired distribution (Ch. 10, we won t cover) Sampling methods: represent desired distribuion with a set of samples, as more samples are used, obtain more accurate representation (Ch. 11, we will cover) Graphical Models Alireza Ghane / Greg Mori 98

Conclusion Readings: Ch. 8 Graphical models depict conditional independence assumptions Directed graphs (Bayesian networks) Factorization of joint distribution as conditional on node given parents Undirected graphs (Markov random fields) Factorization of joint distribution as clique potential functions Inference algorithm sum-product, based on local message passing Works for tree-structured graphs Non-tree-structured graphs, either slow exact or approximate inference Graphical Models Alireza Ghane / Greg Mori 99