Graphical Models. Lecture 3: Local Condi6onal Probability Distribu6ons. Andrew McCallum

Transcription

1 Graphical Models Lecture 3: Local Condi6onal Probability Distribu6ons Andrew McCallum Thanks to Noah Smith and Carlos Guestrin for some slide materials. 1

2 Condi6onal Probability Distribu6ons Proper CPD: x Val(X) P (X = x Parents(X)) = 1 (for con6nuous case, integral) Everything breaks down into distribu6ons over one variable given some others. 2

3 Condi6onal Probability Distribu6ons So far, we ve seen table representa6ons. How many parameters? Where will they come from? Alterna6ves? If we know more about the form of a CPD, we may be able to infer more proper6es of P. Independence asser6ons Efficient inference (later) 3

4 Determinis6c CPDs Some6mes a variable s value is a determinis6c func6on of its parents values. allelle from Mom G 1 G 2 allelle from Dad T phenotypical blood type P(T G 1, G 2 ) G i = a and G 2- i = b G i = a and G 2- i {a, o} G i = b and G 2- i {b, o} G 1 = G 2 = o ab a b o

5 Determinis6c CPDs Affect I(G) G 1 G 2 T X Y T (local Markov assump6on) X Y {G 1, G 2 } X Y Can we derive such independence asser6ons? 5

6 Review: D- Separa6on Three sets of nodes: X, Y, and observed nodes Z X and Y are d- separated given Z if there is no ac6ve trail from any X X to any Y Y given Z. 6

7 D- Separa6on with Determinis6c CPDs Query: X Y Z? Determinis6c variables: D Algorithm: Let Z = Z While there is an X i D such that Parents(X i ) Z, add X i to Z Calculate d- separa6on between X and Y given Z This is sound and complete. But if we know s"ll more about the determinis6c func6ons involved, we may be able to go farther! 7

8 Another Example D A B Suppose C = A XOR B In the case of XOR, B and C fully determine A E C D E {B, C} 8

9 Context- Specific Independence A X C B Y Suppose C = A OR B We are given A = 1. This implies that C = 1. P(X B, A=1) = P(X A=1) Independence! This does not work if A = 0. Previously we defined X Y Z to represent the assump6on P(X Y, Z) = P(X Z) for all values of X, Y, Z. Determinis6c func6ons can imply a type of independence that holds only for par6cular values of some variables. 9

10 Context- Specific Independence Let X, Y, and Z be disjoint sets of variables; C is a set of variables that could overlap. Let c Val(C). X and Y are condi6onally independent given Z and the context c: X c Y Z Defini6on P(X Y, Z, C = c) = P(X Z, C = c) whenever P(Y, Z, C = c) > 0 10

11 Tree CPDs Use a tree to represent P(X Parents(X)) Each leaf is a distribu6on over X Each path defines a context 11

12 Student Example intelligence I difficulty of the course D apply A standardized test score S L G leoer grade job offered J P(J A, S, L) yes no 12

13 Student Example If you don t apply for the job, the outcome does not depend on your score or leoer. intelligence I difficulty of the course D apply A standardized test score S L G leoer grade job offered J P(J A, S, L) 0?? yes no 13

14 Student Example If your test scores are high enough, the recruiter doesn t even look at the leoer. intelligence I difficulty of the course D apply A standardized test score S L G leoer grade job offered J P(J A, S, L) 0?? * yes no 14

15 Student Example 0 A 1 0 S 1 0 L 1 P(J A, S, L) 0?? * yes no 15

16 Other Structured CPDs Rules CPD trees can always be represented compactly as rules, but the converse does not hold Decision diagrams Any kind of par66on structure of Val(X) Val(Parents(X)) Context- specific CPDs can make some edges spurious (given the context)! 16

17 Independent Causes A different kind of CPD structure Consider random variable Y and its parents, Parents(Y) = X Two examples Noisy OR Generalized linear models 17

18 Another Professor (Perfect World) Leoer quality depends on whether you par6cipated by asking good ques6ons (Q), and on whether you wrote a good final paper (F) Q L F P(L Q, F) Q = 1 and F = 1 Q = 1 and F = 0 Q = 0 and F = 1 Q = 0 and F = 0 high low

19 Another Professor (Real World) Leoer quality depends on whether you par6cipated by asking good ques6ons (Q), and on whether you wrote a good final paper (F) Q L F P(L Q, F) Q = 1 and F = 1 Q = 1 and F = 0 Q = 0 and F = 1 Q = 0 and F = 0 high low

20 Another Professor (Real World) P(Q Q) Q = 1 Q = 0 high low Q F Q L F L = Q F P(F F) F = 1 F = 0 high low

21 Another Professor P(Q Q) Q = 1 Q = 0 high low Q F Q L F noise parameter, λ i L = Q F P(F F) F = 1 F = 0 high low

22 Another Professor P(Q Q) Q = 1 Q = 0 high low Q F leak probability, λ 0 Q L F 1 ε ε Leak L = Q F Leak 0 1 P(F F) F = 1 F = 0 high low

23 Noisy OR Model λ i is the noise parameter for X i. P (Y =0 X) = (1 λ 0 ) P (Y =1 X) = 1 i:x i =1 (1 λ 0 ) (1 λ i ) i:x i =1 (1 λ i ) Or equivalently wrioen (where x 1 = 1 and x 0 = 0) k P (y 0 x 1,..., x k ) = (1 λ 0 ) (1 λ i ) x i i=1 23

24 Noisy OR as a Condi6onal Bayesian Network X 1 X 2 X n X 1 X 2 X n Y i, j: X X Y 24

25 (What is a Condi6onal Bayesian Network?) Condi6onal Bayesian Network is a BN with three types of variables: Inputs, always observed, no parents: X Outputs: Y Encapsulated: Z P (Y, Z X) = W Y Z P (W Parents(W )) X Z Y 25

26 Independent Causes Many addi6ve effects combine to score X P(Y = 1) is defined as a func6on of X sigmoid(score(x)) score(x) sigmoid(z) = e z 1+e 26 z

27 Generalized Linear Model Score is defined as a linear func6on of X: Z = f(x) is a random variable! f(x) =w 0 + w i X i i Z Probability distribu6on over binary value Y is commonly* defined by: P (Y = 1) = sigmoid(f(x)) Logis6c regression Maximum entropy classifier sigmoid(z) = e z 1+e z * not the only choice aka logit function 27

28 Logis6c CPD Model as a Condi6onal Bayesian Network X 1 X 2 X n Z sum Y sigmoid Compare: Naïve Bayes model 28

29 Logis6c Models Weight w i can be posi6ve or nega6ve. The X and Y do not need to be binary. Very useful: mul6nomial logis6c to allow many values for Y; indicator variables to allow many values for X Mul6nomial logit 29

30 CPDs with Causal Independence Captures Noisy- OR and Generalized linear models X 1 X 2 X n individual noise models Z 1 Z 2 Z n determinis6c and symmetrically decomposable (binary, commuta6ve, associa6ve opera6on on the Z i ), then Symmetric Independence of Causal Influence (symmetric ICI) Noisy OR: X Z i = simple noisy model Z i s Z = OR Z Y = copy Z Y All interac6on among X i happens here! GLM: X Z i = w i X i Z i s Z = sum Z Y = sigmoid 30

31 Something Interes6ng Happened! We are now represen6ng condi6onal probability distribu6ons as condi6onal Bayesian Networks! 31

32 X 1 X 2 X n Z 1 Z 2 Z n Z Y 32

33 X 1 X 2 X n Z 1 Z 2 Z n Z Y P(Y X)

34 Con6nuous Random Variables First case: con6nuous child and parents Gaussian distribu6on is a CPD: P(Y Mean = μ, Variance = σ 2 ) = Normal(μ, σ 2 ) Linear Gaussian CPD: Normal(linear(x), σ 2 ) Y is a linear func6on of the variables X, with Gaussian noise that has variance σ 2 34

35 Example X (t) is a vehicle s posi6on at 6me t V (t) is its velocity at 6me t X (t+1) X (t) + V (t) Allow for some randomness in the mo6on: V (t) Gaussian mean X (t) Z Gaussian distribu6on X (t+1) Linear Gaussian Model p(y x 1,...x k )=N (β 0 + β 1 x β k x k ; σ 2 ) 35

36 Con6nuous Random Variables Second case: con6nuous child with discrete and con6nuous parents (X disc and X cont ) Condi6onal linear Gaussian : P (Y X disc = x disc, X cont = x cont )=N w xdisc,0 + j w xdisc,j x cont,j, σx 2 disc different weights for each x disc Induces a Gaussian mixture for Y 36

37 Con6nuous Random Variables Third case: discrete child Y with con6nuous parent X Threshold model Makes P(Y X) discon6nuous in X s value Mul6nomial logit (see slide 27 Generalized Linear Model ) 37

38 CPDs can be Bayesian Networks! Condi"onal Bayesian Network is a BN with three types of variables: Inputs, always observed, no parents: X Outputs: Y Encapsulated: Z Condi6onal Probability Distribu6on P (Y, Z X) = W Y Z P (W Parents(W )) A CPD is an Encapsulated CPD if it can be represented by a Condi"onal Bayesian Network. Construct a complex BN, with components composed of other BN subcomponents!...object- oriented style! 38

39 Encapsulated CPDs (K&F Figure 5.15) 39

40 Where We Are Structure in CPDs Effects on independence asser6ons Examples: Determinism Context- specificity Independent causes Con6nuous distribu6ons CPDs represented by Condi6onal BNs 40