Probabilistic Graphical Models Homework 1: Due January 29, 2014 at 4 pm

Transcription

1 Probabilistic Grapical Models Homework 1: Due January 29, 2014 at 4 pm Directions. Tis omework assignment covers te material presented in Lectures 1-3. You must complete all four problems to obtain full credit. To submit your assignment, please upload a pdf file containing your writeup and a zip file containing your code to Canvas by 4 pm on Wednesday, January 29t. We igly encourage tat you type your omework using te L A TEXtemplate provided on te course website, but you may also write it by and and ten scan it. 1 Fundamentals [25 points] Tis question will refer to te grapical models sown in Figures 1 and 2, wic encode a set of independencies among te following variables: Season (S), Flu (F), Deydration (D), Cills (C), Headace (H), Nausea (N), Dizziness (Z). Note tat te two models ave te same skeleton, but Figure 1 depicts a directed model (Bayesian network) wereas Figure 2 depicts an undirected model (Markov network). Season Flu Deydration Cills Headace Nausea Dizziness Figure 1: A Bayesian network tat represents a joint distribution over te variables Season, Flu, Deydration, Cills, Headace, Nausea, and Dizziness. Part 1: Independencies in Bayesian Networks [12 points] Consider te model sown in Figure 1. Indicate weter te following independence statements are true or false according to tis model. Provide a very brief justification of your answer (no more tan 1 sentence). 1. Season Cills False: influence can flow along te pat Season Flu Cills, since Flu is unobserved 1

2 2. Season Cills Flu True: influence cannot flow troug Flu, since it is observed; tere are no oter pats linking Season and Cills 3. Season Headace Flu False: influence can flow along te pat Season Deydration Headace, since Deydration is unobserved 4. Season Headace Flu, Deydration True: since bot Flu and Deydration are observed, influence cannot flow along any pat tat links Season and Headace 5. Season Nausea Deydration False: influence can flow along te pat formed by Season Flu Headace Dizziness Nausea, since Flu, Headace, and Dizziness are unobserved 6. Season Nausea Deydration, Headace True: influence cannot flow along te pat Season Deydration Nausea, since Deydration is observed; influence cannot flow along te pat Season Flu Headace Dizziness Nausea, since Headace is observed; influence cannot flow along te pat Season Flu Headace Deydration Nausea, even toug tere is an observed v-structure centered at Headace, because Deydration is observed 7. Flu Deydration False: influence can flow along te pat Flu Season Deydration, since Season is unobserved 8. Flu Deydration Season, Headace False: influence can flow along te pat Flu Headace Deydration, since tis is a v-structure and Headace is observed 9. Flu Deydration Season True: influence cannot flow troug Season, wic is observed, nor troug Headace or Nausea, since bot form v-structures and bot are unobserved 10. Flu Deydration Season, Nausea False: influence can flow along te pat Flu Headace Dizziness Nausea Deydration, since Headace and Dizziness are unobserved and tere is a v-structure at Nausea, wic is observed 11. Cills Nausea False: influence can flow along te pat Cills Flu Season Deydration Nausea, since Flu, Season, and Deydration are all unobserved 12. Cills Nausea Headace False: influence can flow along te pat Cills Flu Headace Deydration Nausea, since tere is a v-structure at Headace, wic is observed Part 2: Factorized Joint Distributions [4 points] 1. Using te directed model sown in Figure 1, write down te factorized form of te joint distribution over all of te variables, P (S, F, D, C, H, N, Z). 2

3 P (S, F, D, C, H, Z, N) P (S) P (F S) P (D S) P (C F ) P (H F, D) P (Z H) P (N D, Z) 2. Using te undirected model sown in Figure 2, write down te factorized form of te joint distribution over all of te variables, assuming te model is parameterized by one factor over eac node and one over eac edge in te grap. 1 Z φ 1(S) φ 2 (F ) φ 3 (D) φ 4 (C) φ 5 (H) φ 6 (N) φ 7 (Z) φ 8 (S, F ) φ 9 (S, D) φ 10 (F, C) φ 11 (F, H) φ 12 (D, H) φ 13 (D, N) φ 14 (H, Z) φ 15 (N, Z) Part 3: Evaluating Probability Queries [7 points] Assume you are given te conditional probability tables listed in Table 1 for te model sown in Figure 1. Evaluate eac of te probabilities queries listed below, and sow your calculations. 1. Wat is te probability tat you ave te flu, wen no prior information is known? Tis translates to P (Flu true) P (F true) s P (F true, S s) s P (F true S s)p (S s) P (F true S wint)p (S wint) + P (F true S summ)p (S summ) Wat is te probability tat you ave te flu, given tat it is winter? P (S winter) P (S summer) P (F true S) P (F false S) S winter S summer P (D true S) P (D false S) S winter S summer P (C true F ) P (C false F ) F true F false P (H true F, D) P (H false F, D) F true, D true F true, D false F false, D true F false, D false P (Z true H) P (Z false H) H true H false P (N true D, Z) P (N false D, Z) D true, Z true D true, Z false D false, Z true D false, Z false Table 1: Conditional probability tables for te Bayesian network sown in Figure 1. 3

4 Tis translates to P (Flu true Season winter) P (F true S wint) Wat is te probability tat you ave te flu, given tat it is winter and tat you ave a eadace? Tis translates to P (Flu true Season winter, Headace true) P (F true S wint, H true) P (F true, S wint, H true) P (S wint, H true) d P (F true, S wint, H true, D d) f,d P (F f, S wint, H true, D d) d P (H true F true, D d)p (F true S wint)p (D d S wint)p (S wint) f,d P (H true F f, D d)p (F f S wint)p (D d S wint)p (S wint) Wat is te probability tat you ave te flu, given tat it is winter, you ave a eadace, and you know tat you are deydrated? Tis translates to P (Flu true Season winter, Headace true, Deydration true) P (F true S wint, H true, D true) P (F true, S wint, H true, D true) P (S wint, H true, D true) P (F true, S wint, H true, D true) f P (F f, S wint, H true, D true) P (H true F true, D true)p (F true S wint)p (D true S wint)p (S wint) f P (H true F f, D true)p (F f S wint)p (D true S wint)p (S wint) Does knowing you are deydrated increase or decrease your likeliood of aving te flu? Intuitively, does tis make sense? Knowing tat you are deydrated decreases te likeliood tat you ave te flu. Tis makes sense because te eadace symptom is explained away by te deydration. Part 4: Bayesian Networks vs. Markov Networks [2 points] Now consider te undirected model sown in Figure Are tere any differences between te set of marginal independencies encoded by te directed and undirected versions of tis model? If not, state te full set of marginal independencies encoded by bot models. If so, give one example of a difference. Tere are no differences, because neiter model encodes any marginal independencies at all. 2. Are tere any differences between te set of conditional independencies encoded by te directed and undirected versions of tis model? If so, give one example of a difference. Tere are several differences. One example is tat in te Markov network, we ave Flu Deydration Season, Headace. However, tis is not te case in te Bayesian network because observing Headace creates an active v-structure at Flu Headace Deydration. 4

5 Season Flu Deydration Cills Headace Nausea Dizziness Figure 2: A Markov network tat represents a joint distribution over te variables Season, Flu, Deydration, Cills, Headace, Nausea, and Dizziness. 2 Bayesian Networks [25 points] Part 1: Constructing Bayesian Networks [8 points] In tis problem you will construct your own Bayesian network (BN) for a few different modeling scenarios described as word problems. By standard convention, we will use saded circles to represent observed quantities, clear circles to represent random variables, and uncircled symbols to represent distribution parameters. In order to do tis problem, you will first need to understand plate notation, wic is a useful tool for drawing large BNs wit many variables. Plates can be used to denote repeated sets of random variables. For example, suppose we ave te following generative process: Draw Y Normal(µ, Σ) For m 1,..., M: Draw X m Normal(Y, Σ) Tis BN contains M + 1 random variables, wic includes M repeated variables X 1,..., X M tat all ave Y as a parent. In te BN, we draw te repeated variables by placing a box around a single node, wit an index in te box describing te number of copies; we ve drawn tis in Figure 3. Y Xm m 1,...,M Figure 3: An example of a Bayesian network drawn wit plate notation. 5

6 For eac of te modeling scenarios described below, draw a corresponding BN. Make sure to label your nodes using te variable names given below, and use plate notation if necessary. 1. (Gaussian Mixture Model). Suppose you want to model a set of clusters witin a population of N entities, X 1,..., X N. We assume tere are K clusters θ 1,..., θ K, and tat eac cluster represents a vector and a matrix, θ k {µ k, Σ k }. We also assume tat eac entity X n belongs to one cluster, and its membersip is given by an assignment variable Z n {1,..., K}. Here s ow te variables in te model relate. Eac entity X n is drawn from a so-called mixture distribution, wic in tis case is a Gaussian distribution, based on its individual cluster assignment and te entire set of clusters, written X n Normal(µ Zn, Σ Zn ). Eac cluster assignment Z n as a prior, given by Z n Categorical(β). Finally, eac cluster θ k also as a prior, given by θ k Normal-invWisart(µ 0, λ, Φ, ν) Normal(µ 0, 1 λσ) invwisart(φ, ν). 2. (Bayesian Logistic Regression). Suppose you want to model te underlying relationsip between a set of N input vectors X 1,..., X N and a corresponding set of N binary outcomes Y 1,..., Y N. We assume tere is a single vector β wic dictates te relationsip between eac input vector and its associated output variable. In tis model, eac output is drawn wit Y n Bernoulli(invLogit(X n β)). Additionally, te vector β as a prior, given by β Normal(µ, Σ). Te correct grapical models are sown below. Note tat for Bayesian logistic regression, it s also correct to draw {X n } as a set of fixed parameters since tey are tecnically not random variables. β µ 0, λ, Φ, ν µ, Σ Z n θ k X n β k 1,,K X n Y n n 1,,N n 1,,N Gaussian Mixture Model Bayesian Logistic Regression Part 2: Inference in Bayesian Networks [12 points] In tis problem you will derive formulas for inference tasks in Bayesian networks. Consider te Bayesian network given in Figure 4. X3 X4 X5 X2 X6 X1 Figure 4: A Bayesian network over te variables X 1,..., X 6. Note tat X 1 is observed (wic is denoted by te fact tat it s saded in) and te remaining variables are unobserved. 6

7 For eac of te following questions, write down an expression involving te variables X 1,..., X 6 tat could be computed by directly plugging in teir local conditional probability distributions. First, give expressions for te following tree posterior distributions over a particular variable given te observed evidence X1 x1. 1. P (X 2 x 2 X 1 x 1 ) P (X 1 x 1 X 2 x 2 ) X 3 X 4 P (X 2 x 2 X 3, X 4 )P (X 3 )P (X 4 ) X 2 P (X 1 x 1 X 2 ) X 3 X 4 P (X 2 X 3, X 4 )P (X 3 )P (X 4 ) 2. P (X 3 x 3 X 1 x 1 ) P (X 3 x 3 ) X 2 P (X 1 x 1 X 2 ) X 4 P (X 2 X 3 x 3, X 4 )P (X 4 ) X 2 P (X 1 x 1 X 2 ) X 3 X 4 P (X 2 X 3, X 4 )P (X 3 )P (X 4 ) 3. P (X 5 x 5 X 1 x 1 ) X 2 P (X 1 x 1 X 2 ) X 3 X 4 P (X 3 )P (X 4 )P (X 5 x 5 X 3 )P (X 2 X 3, X 4 ) X 2 P (X 1 x 1 X 2 ) X 3 X 4 P (X 2 X 3, X 4 )P (X 3 )P (X 4 ) Second, give expressions for te following tree conditional probability queries. Note tat tese types of expressions are useful for te inference algoritms tat we ll learn later in te class. 4. P (X 2 x 2 X 1 x 1, X 3 x 3, X 4 x 4, X 5 x 5, X 6 x 6 ) P (X 1 x 1 X 2 x 2 )P (X 2 x 2 X 3 x 3, X 4 x 4 )P (X 3 x 3 )P (X 4 x 4 ) X 2 P (X 1 x 1 X 2 )P (X 2 X 3 x 3, X 4 x 4 )P (X 3 x 3 )P (X 4 x 4 ) 5. P (X 3 x 3 X 1 x 1, X 2 x 2, X 4 x 4, X 5 x 5, X 6 x 6 ) P (X 2 x 2 X 3 x 3, X 4 x 4 )P (X 5 x 5 X 3 x 3 )P (X 3 x 3 )P (X 4 x 4 ) X 3 P (X 2 x 2 X 3, X 4 x 4 )P (X 5 x 5 X 3 )P (X 3 )P (X 4 x 4 ) 6. P (X 5 x 5 X 1 x 1, X 2 x 2, X 3 x 3, X 4 x 4, X 6 x 6 ) P (X 5 x 5 X 3 x 3 ) Part 3: On Markov Blankets [5 points] In tis problem you will prove a key property of Markov blankets in Bayesian networks. Recall tat te Markov blanket of a node in a BN consists of te node s cildren, parents, and coparents (i.e. te cildren s oter parents). Also recall tat tere are four basic types of two-edge trails in a BN, wic are illustrated in Figure 5: te causal trail (ead-to-tail), evidential trail (tail-to-ead), common cause (tail-to-tail), and common effect (ead-to-ead). Causal Trail Evidential Trail Common Cause Common Effect Figure 5: Illustration of te four basic types of two-edge trails in a BN. 7

8 Using te four trail types, prove te following property of BNs: given its Markov blanket, a node in a Bayesian network is conditionally independent of every oter set of nodes. Proof: Te Markov blanket for a node X consists of its cildren, parents, and coparents. Assume tat we condition on te Markov blanket of X. In order for X to be conditionally dependent on any oter set of nodes S, tere must exist an active trail between X and a member of S. We will sow tere does not exist an active trail. First, te active trail cannot include te edge between X and any of its parents. If it did, tis would imply eiter an evidential trail or a common cause (starting at node X, going troug te parent), and bot of tese do not yield an active trail wen te parent is conditioned upon. Secondly, te active trail cannot include te edge between X and any of its cildren. Tis is because it would imply eiter a casual trail or a common effect. In te first case, a casual trail (starting at node X, going troug te cild) would not yield an active trail wen te cild is conditioned upon. In te second case, a common effect (starting at node X, going troug te cild, and ending at a coparent) would yield an active trail; owever, tis implies eiter an evidential trail or a common cause (starting at te cild of X, going troug te coparent), and bot of tese do not yield an active trail wen te coparent is conditioned upon. Terefore, in all cases, X is d-separated from any set S given its Markov blanket, and is terefore conditionally independent of any set S. 3 Restricted Boltzmann Macines [25 points] Restricted Boltzmann Macines (RBMs) are a class of Markov networks tat ave been used in several applications, including image feature extraction, collaborative filtering, and recently in deep belief networks. An RBM is a bipartite Markov network consisting of a visible (observed) layer and a idden layer, were eac node is a binary random variable. One way to look at an RBM is tat it models latent factors tat can be learned from input features. For example, suppose we ave samples of binary user ratings (like vs. dislike) on 5 movies: Finding Nemo (V 1 ), Avatar (V 2 ), Star Trek (V 3 ), Aladdin (V 4 ), and Frozen (V 5 ). We can construct te following RBM: Figure 6: An example RBM wit 5 visible units and 2 idden units. Here, te bottom layer consists of visible nodes V1,..., V5 tat are random variables representing te binary ratings for te 5 movies, and H1, H2 are two idden units tat represent latent factors to be learned during training (e.g., H1 migt be associated wit Disney movies, and H2 could represent te adventure genre). If we are using an RBM for image feature extraction, te visible layer could instead denote binary values associated wit eac pixel, and te idden layer would represent te latent features. However, for tis problem we will stick wit te movie example. In te following questions, let V (V1,..., V5) be a vector of ratings (e.g. te observation v (1, 0, 0, 0, 1) implies tat a user likes only Finding Nemo and Aladdin). Similarly, let H (H1, H2) be a vector of latent factors. Note tat all te random variables are binary and take on states in {0, 1}. Te joint distribution of a configuration is given by p(v v, H ) 1 Z e E(v,) (1) 8

9 were E(v, ) ij w ij v i j i a i v i j b j j is te energy function, {w ij }, {a i }, {b j } are model parameters, and Z Z({w ij }, {a i }, {b i }) v, e E(v,) is te partition function, were te summation runs over all joint assignments to V and H. 1. [7 pts] Using Equation (1), sow tat p(h V ), te distribution of te idden units conditioned on all of te visible units, can be factorized as p(h V ) j p(h j V ) (2) were p(h j 1 V v) σ ( b j + i w ij v i ) and σ(s) es 1+e is te sigmoid function. Note tat p(h s j 0 V v) 1 p(h j 1 V v). p(v, ) p(v, ) p(h V v) p(v) p(v, ) exp ( ( i a iv i ) exp ij w ijv i j + ) j b j j exp ( i a iv i ) ( exp ij w ijv i j + ) j b j j j exp ( i w ijv i j + b j j ) j exp ( i w ijv i j + b j j ) j exp ( i w ijv i j + b j j ) j j exp ( i w ijv i j + b j j ) exp ( i w ijv i j + b j j ) 1 + exp ( j i w ijv i + b j ) j p( j v) and tus p(h j 1 V v) σ( i w ijv i +b j ). Note during te derivation te sum and product excanges in te denominator because j exp ( i w ijv i j + b j j ) 1... n f( 1 )...f( n ) were f( j ) exp( i w ijv i j + b j j ) so te sum can be pused into products. 2. [3 pts] Give te factorized form of p(v H), te distribution of te visible units conditioned on all of te idden units. Tis sould be similar to wat s given in part 1, and so you may omit te derivation. By symmetry, we ave p(v H) i p(v i H) and p(v i 1 H ) σ a i + j w ij j 9

10 3. [2 pts] Can te marginal distribution over idden units p(h) be factorized? If yes, give te factorization. If not, give te form of p(h) and briefly justify. No. Te form of p(h) is given by: p() v p(v, ) v exp ( E(v, )) exp b j j v ij w ij v i j + i a i v i + j exp b j j exp a i v i j v i,j w ij v i j + i exp j b j j v exp i j w ij v i j + a i v i j j j exp(b j j ) v exp(b j j ) i exp(b j j ) i exp v i w ij j + a i i j exp v i w ij j + a i v i j 1 + exp w ij j + a i j Since te second term is a product over visible units i, not idden unit j, p() does not factorize. 4. [4 pts] Based on your answers so far, does te distribution in Equation (1) respect te conditional independencies of Figure (6)? Explain wy or wy not. Are tere any independencies in Figure 6 tat are not captured in Equation (1)? Since RBM is a full bipartite grap (all nodes are connected from te nodes on te oter side), te only indepencies implied by te grap are te ones sown in part 1 and 2. Tus te answer to te two parts are Yes and No. 5. [7 pts] We can use te log-likeliood of te visible units, log p(v v), as te criterion to learn te model parameters {w ij }, {a i }, {b j }. However, tis maximization problem as no closed form solution. One popular tecnique for training tis model is called contrastive divergence and uses an approximate gradient descent metod. Compute te gradient of te log-likeliood objective wit respect to w ij by sowing te following: log p(v v) w ij p(h V v)v i j v, p(v v, H )v i j E [V i H j V v] E [V i H j ] were E [V i H j V v] can be readily evaluated using Equation (2), but E [V i H j ] is tricky as te expectation is taken over not just H j but also V i. Hint 1: To save some writing, do not expand E(v, ) until you ave E(v,) w ij. Hint 2: Te partition function, Z, is a function of w ij. 10

11 log p(v) w ij log w ij log w ij p(v, ) exp( E(v, )) Z Z exp( E(v, )) Z exp( E(v, )) w ij ( 1 Z exp( E(v, )) Z exp( E(v, )) E(v, ) exp( E(v, )) w ij p( v)v i j + v, w ij exp( E(v, )) exp( E(v, )) E(v, ) Z w ij exp( E(v, )) Z 2 1 exp( E(v, )) Z w ij v, Z w ij ) p( v)v i j v, p(v, )v i j 6. [2 pts] After training, suppose H 1 1 corresponds to Disney movies, and H 2 1 corresponds to te adventure genre. Wic w ij do you expect to be positive, were i indexes te visible nodes and j indexes te idden nodes? List all of tem. w 11, w 41, w 51, w 22, w 32 4 Image Denoising [25 points] Tis is a programming problem involving Markov networks (MNs) applied to te task of image denoising. Suppose we ave an image consisting of a 2-dimensional array of pixels, were eac pixel value Z i is binary, i.e. Z i {+1, 1}. Assume now tat we make a noisy copy of te image, were eac pixel in te image is flipped wit 10% probability. A pixel in tis noisy image is denoted by X i. We sow te original image and te image wit 10% noise in Figure 7. Given te observed array of noisy pixels, our goal is to recover te original array of pixels. To solve tis problem, we model te original image and noisy image wit te following MN. We ave a latent variable Z i for eac noise-free pixel, and an observed variable X i for eac noisy pixel. Eac variable Z i as an edge leading to its immediate neigbors (to te Z i associated wit pixels above, below, to te left, and to te rigt, wen tey exist). Additionally, eac variable Z i as an edge leading to its associated observed pixel X i. We illustrate tis MN in Figure 8. Denote te full array of latent (noise-free) pixels as Z and te full array of observed (noisy) pixels as X. We define te energy function for tis model as E(Z z, X x) i z i β {i,j} z i z j ν i z i x i (3) were te first and tird summations are over te entire array of pixels, te second summation is over all pairs of latent variables connected by an edge, and R, β R +, and ν R + denote constants tat must be cosen. Using te binary image data saved in w1 images.mat, your task will be to infer te true value of eac pixel (+1 or 1) by optimizing te above energy function. To do tis, initialize te Z i s to teir noisy values, and ten iterate troug eac Z i and ceck weter setting it s value to +1 or 11

12 Figure 7: Te original binary image is sown on te left, and a noisy version of te image in wic a randomly selected 10% of te pixels ave been flipped is sown on te rigt. xij zij Figure 8: Illustration of te Markov network for image denoising. 1 yields a lower energy (iger probability). Repeat tis process, making passes troug all of te pixels, until te total energy of te model as converged. You must specify values of te constants R, β R +, and ν R +. Report te error rate (fraction of pixels recovered incorrectly) tat you acieve by comparing your denoised image to te original image tat we provide, for tree different settings of te tree constants. Include a figure of your best denoised image in your writeup. Also make sure to submit a zipped copy of your code. Te TAs will give a special prize to te student wo is able to acieve te lowest error on tis task. Hint 1: Wen evaluating weter +1 or 1 is a better coice for a particular pixel Z i, you do not need to evaluate te entire energy function, as tis will be computationally very expensive. Instead, just compute te contribution by te terms tat are affected by te value of Z i. Hint 2: If you d like to try and compete to acieve te best performance, you can work to find good parameters, or even modify te algoritm in an intelligent way (be creative!). However, if you come up wit a modified algoritm, you sould separately report te new error rate you acieve, and also turn in a second.m file (placed in your zipped code directory) wit te modified algoritm. For a solution, please see te code in te zipped directory. 12