STATISTICAL METHODS IN AI/ML Vibhav Gogate The University of Texas at Dallas. Bayesian networks: Representation

Transcription

1 STATISTICAL METHODS IN AI/ML Vibhav Gogate The University of Texas at Dallas Bayesian networks: Representation

2 Motivation Explicit representation of the joint distribution is unmanageable Computationally: Memory intensive to store and manipulate Cognitively: Impossible to acquire so many numbers from human experts Statistically: We will need ridiculously large amount of data to learn. Solution: Exploit Independence properties and Represent the distribution using a graph Trouble: Mapping the logic of probability theory into graph theory!

3 Properties of Independence The statement I(X, Z, Y) means that X is independent of Y given Z. Namely, Pr(X Y, Z) = Pr(X Z) and Pr(X, Y Z) = Pr(X Z) Pr(Y Z) Symmetry I(X, Z, Y) I(Y, Z, X) Decomposition I(X, Z, Y W) I(X, Z, Y) Weak Union I(X, Z, Y W) I(X, Z W, Y) Contraction I(X, Z Y, W)&I(X, Z, Y) I(X, Z, Y W) Intersection For any positive distribution: I(X, Z W, Y)&I(X, Z Y, W) I(X, Z, Y W)

4 Proof of Symmetry Assume that I(X, Z, Y) holds. This implies that: Pr(X, Y Z) = Pr(X Z) Pr(Y Z) (1) i.e. I(Y, Z, X) holds too (exchanging the positions of X and Y).

5 Proof of Decomposition Assume that I(X, Z, Y W) holds. Then, Pr(X, Y, W Z) = Pr(X Z) Pr(Y, W Z) Pr(X, Y Z) = w = w Pr(X, Y, w Z) (2) Pr(X Z) Pr(Y, w Z) (3) = Pr(X Z) w Pr(Y, w Z) (4) i.e. I(X, Z, Y) holds too. = Pr(X Z) Pr(Y Z) (5)

6 Proof of Weak Union Assume that I(X, Z, Y W) holds. Then, Pr(X, Y, W Z) = Pr(X Z) Pr(Y, W Z) Pr(X, Y W, Z) = Pr(X, Y, W Z) Pr(W Z) (6) = Pr(X Z) Pr(Y, W Z) Pr(W Z) (7) = Pr(X Z) Pr(Y W, Z) (8) I stopped here: Homework problem Which of the previous properties can you use to prove that: Pr(X, Y W, Z) = Pr(X W, Z) Pr(Y W, Z)

7 Bayesian networks: Directed-Graphs Mapping a distribution to a Graph!! The graph can be viewed in two different ways: As a data structure to represent the joint distribution compactly As a compact representation of a set of conditional independence assumptions about a distribution The two views are equivalent

8 Bayesian networks: Data Structure view Bayesian network = Use Chain rule + Conditional Independence properties Chain rule: P(X 1,..., X n ) = n P(X i X 1,..., X i 1 ) i=1

9 Bayesian networks: Data Structure view Chain rule: P(X 1,..., X n ) = n i=1 P(X i X 1,..., X i 1 ) Use the chain rule to represent the joint distribution rather than a giant table! Example Intelligence I and SAT Score S P(I, S) = P(I)P(S I) s 0 s 1 i * *0.7 i 1 0.2* *0.3 = i0 i s0 s1 i i

10 Bayesian networks: Data Structure view However, we don t gain anything by using the chain rule. Space complexity is the same. Exploit conditional independence properties What if I tell you that you are representing a joint distribution over 2 coin tosses? Chain rule : P(X 1, X 2 ) = P(X 1 )P(X 2 X 1 ) Conditional Independence : P(X 1, X 2 ) = P(X 1 )P(X 2 )

11 Bayesian networks: Data Structure view Chain rule P(X 1,..., X n ) = n i=1 P(X i X 1,..., X i 1 ) as a directed graph. X 1,..., X i 1 are the parents of X i. Complete Graph If we know that P(X i X 1,..., X i 1 ) = P(X i Y i ) where Y i is a subset of {X 1,..., X i 1 }. Then, we get a sparse graph (a Bayesian network).

12 Bayesian networks: Data structure view d 0 d g 1 i 0,d i 0,d i 0,d i 0,d g 2 g Difficulty Grade Letter l 0 g g g Intelligence l i 0 i 1 SAT s 0 i i s Random variables are nodes and edges represent direct influence of one variable on other Each node is associated with a conditional probability table (CPT). The joint distribution is product of all CPTs P(D, I, G, S, L) = P(D)P(I)P(G D, I)P(L G)P(S I) What is P(i 1, d 0, g 2, s 1, l 0 )?

13 Bayesian networks: Data structure view d 0 d g 1 i 0,d i 0,d i 0,d i 0,d g 2 g Difficulty Grade Letter l 0 g g g Intelligence l i 0 i 1 SAT s 0 i i s Space complexity? Assume each variable has d values in its domain. O(d k+1 ) for each variable having k parents.

14 Graph terms

15 Bayesian networks: Compact Representation of Conditional Independence statements view A directed acyclic graph G represents the following independence statements Markov(G) = I(V, Parents(V ), Non Descendants(V )) Parents(V) denote the direct causes of V and Descendants(V) denote the effects of V Given the direct causes of a variable, our beliefs in that variable become independent of its non-effects.

16 Bayesian networks: Compact Representation of Conditional Independence statements view Markovian assumptions?

17 Bayesian networks: Compact Representation of Conditional Independence statements view

18 Expanding Markov (G) using properties of probabilistic independence Markov(G) is not comprehensive. We can expand it using properties such as symmetry, decomposition, weak-union, contraction and intersection. Given W Non Descendants(X)), how can we strengthen Markov(G) = I(X, Parents(X), Non Descendants(X)) Decomposition: I(X, Z, Y W) I(X, Z, Y)

19 Expanding Markov (G) using properties of probabilistic independence Markov(G) is not comprehensive. We can expand it using properties such as symmetry, decomposition, weak-union, contraction and intersection. Given W Non Descendants(X)), how can we strengthen Markov(G) = I(X, Parents(X), Non Descendants(X)) Decomposition: I(X, Z, Y W) I(X, Z, Y) I(X, Parents(X), W) Weak Union: I(X, Z, Y W) I(X, Z W, Y)

20 Expanding Markov (G) using properties of probabilistic independence Markov(G) is not comprehensive. We can expand it using properties such as symmetry, decomposition, weak-union, contraction and intersection. Given W Non Descendants(X)), how can we strengthen Markov(G) = I(X, Parents(X), Non Descendants(X)) Decomposition: I(X, Z, Y W) I(X, Z, Y) I(X, Parents(X), W) Weak Union: I(X, Z, Y W) I(X, Z W, Y) and so on. I(X, Parents(X) W, Non Decendants(X) \ W)

21 Expanding Markov(G) using Symmetry

22 Expanding Markov(G) using Decomposition

23 Expanding Markov(G) using Weak-union

24 Expanding Markov(G) using Weak-union

25 Graphoid Axioms

26 Capturing independence graphically Question: Is there a purely graphical test that can find all of these independence statements, namely Markov(G) plus the ones inferred using the properties of conditional independence? YES, it is called d-separation.

27 D-separation

28 D-separation

29 D-separation

30 D-separation

31 D-separation

32 D-separation

33 D-separation

34 D-separation

35 D-separation

36 D-separation

37 D-separation

38 D-separation

39 D-separation

40 D-separation

41 D-separation

42 D-separation The d-separation test is sound If distribution Pr is induced by Bayesian network G, then dsep G (X, Z, Y) only if I Pr (X, Z, Y) The proof of soundness is constructive showing that every independence claimed by d-separation can indeed be derived using the graphoid axioms.

43 I-map, D-map and P-map A directed acyclic graph (a Bayesian network) describes a set of conditional independence assumptions I G. It is an I-map of a distribution I Pr if I G I Pr or I G I Pr It is a D-map if I Pr I G or I Pr I G It is a P-map if it is both an I-map and a D-map. Namely, I G = I Pr I-maps and D-maps can be constructed trivially. Therefore, we enforce minimality.

44 Minimal I-maps

45 Minimal I-maps Minimal I-maps are not unique. Different orderings give rise to different I-maps D I D I D I G S G S G S L L L (a) (b) (c) Ordering for (b): (L, S, G, I, D) Ordering for (c): (L, D, S, I, G)

46 Blankets and Boundaries

47 Blankets and Boundaries

48 Blankets and Boundaries