Bayesian Belief Network

Transcription

1 Bayesian Belief Network a! b) = a) b) toothache, catch, cavity, Weather = cloudy) = = Weather = cloudy) toothache, catch, cavity) The decomposition of large probabilistic domains into weakly connected subsets via conditional independence is one of the most important developments in the recent history of AI This can work well, even the assumption is not true! 1

2 v NB Naive Bayes assumption: which gives Bayesian networks Conditional Independence Inference in Bayesian Networks Irrelevant variables Constructing Bayesian Networks Aprendizagem Redes Bayesianas Examples - Exercisos 2

3 Naive Bayes assumption of conditional independence too restrictive But it's intractable without some such assumptions... Bayesian Belief networks describe conditional independence among subsets of variables allows combining prior knowledge about (in)dependencies among variables with observed training data Bayesian networks A simple, graphical notation for conditional independence assertions and hence for compact specification of full joint distributions Syntax: a set of nodes, one per variable a directed, acyclic graph (link "directly influences") a conditional distribution for each node given its parents: P (X i Parents (X i )) In the simplest case, conditional distribution represented as a conditional probability table (CPT) giving the distribution over X i for each combination of parent values 3

4 Bayesian Networks Bayesian belief network allows a subset of the variables conditionally independent A graphical model of causal relationships Represents dependency among the variables Gives a specification of joint probability distribution X Z Y P Nodes: random variables Links: dependency X,Y are the parents of Z, and Y is the parent of P No dependency between Z and P Has no loops or cycles Conditional Independence Once we know that the patient has cavity we do not expect the probability of the probe catching to depend on the presence of toothache catch cavity! toothache) = catch cavity) toothache cavity! catch) = toothache cavity) Independence between a and b a b) = a) b a) = b) 4

5 Example Topology of network encodes conditional independence assertions: Weather is independent of the other variables Toothache and Catch are conditionally independent given Cavity Bayesian Belief Network: An Example Family History Smoker (FH, S) (FH, ~S) (~FH, S) (~FH, ~S) LC LungCancer Emphysema ~LC PositiveXRay Dyspnea Bayesian Belief Networks The conditional probability table for the variable LungCancer: Shows the conditional probability for each possible combination of its parents 5

6 Example I'm at work, neighbor John calls to say my alarm is ringing, but neighbor Mary doesn't call. Sometimes it's set off by minor earthquakes. Is there a burglar? Variables: Burglary, Earthquake, Alarm, JohnCalls, MaryCalls Network topology reflects "causal" knowledge: A burglar can set the alarm off An earthquake can set the alarm off The alarm can cause Mary to call The alarm can cause John to call Belief Networks Burglary B) Earthquake E) Alarm Burg. Earth. A) t t.95 t f.94 f t.29 f f.001 JohnCalls A J) t.90 f.05 MaryCalls A M) t.7 f.01 6

7 Full Joint Distribution P ( x,..., xn) =! x n parents( X )) 1 i i i= 1 j " m " a " b " e) = j a) m a) a b " e) b) e) = 0.9! 0.7! 0.001! 0.999! = Compactness A CPT for Boolean X i with k Boolean parents has 2 k rows for the combinations of parent values Each row requires one number p for X i = true (the number for X i = false is just 1-p) If each variable has no more than k parents, the complete network requires O(n 2 k ) numbers I.e., grows linearly with n, vs. O(2 n ) for the full joint distribution For burglary net, = 10 numbers (vs = 31) 7

8 Inference in Bayesian Networks How can one infer the (probabilities of) values of one or more network variables, given observed values of others? Bayes net contains all information needed for this inference If only one variable with unknown value, easy to infer it In general case, problem is NP hard Example In the burglary network, we migth observe the event in which JohnCalls=true and MarryCalls=true We could ask for the probability that the burglary has occurred Burglary JohnCalls=ture,MarryCalls=true) 8

9 Remember - Joint distribution cavity toothache) = = cavity toothache) = cavity " toothache) toothache) = 0.6 = cavity " toothache) toothache) = 0.4 Normalization 1 = y x) + y x) Y X ) = "! X Y ) Y ) " y x), y x) " 0.12,0.08 = 0.6,0.4 9

10 Normalization Cavity toothache) = "Cavity, toothache) = "[Cavity, toothache,catch) + Cavity, toothache, catch)] = "[< 0.108,0.016 > + < 0.012,0.064 >] = " < 0.12,0.08 >=< 0.6,0.4 > X is the query variable E evidence variable Y remaining unobservable variable X e) = "X,e) = " # X,e,y) Summation over all possible y (all possible values of the unobservable variables Y) y Burglary JohnCalls=ture,MarryCalls=true) The hidden variables of the query are Earthquake and Alarm B j,m) = "B, j,m) = " ## e B,e,a, j,m) For Burglary=true in the Bayesain network a ## b j,m) = " b)e)a b,e) j a)m a) e a 10

11 To compute we had to add four terms, each computed by multiplying five numbers In the worst case, where we have to sum out almost all variables, the complexity of the network with n Boolean variables is O(n2 n ) b) is constant and can be moved out, e) term can be moved outside summation a # b j,m) = "b) e) a b,e) j a)m a) e # a JohnCalls=true and MarryCalls=true, the probability that the burglary has occured is aboud 28% B, j,m) = " < , >#< 0.284,0.716 > 11

12 Computation for Burglary=true Variable elimination algorithm Eliminate repeated calculation Dynamic programming 12

13 Irrelevant variables (X query variable, E evidence variables) Complexity of exact inference The burglary network belongs to a family of networks in which there is at most one undirected path between tow nodes in the network These are called singly connected networks or polytrees The time and space complexity of exact inference in polytrees is linear in the size of network Size is defined by the number of CPT entries If the number of parents of each node is bounded by a constant, then the complexity will be also linear in the number of nodes 13

14 For multiply connected networks variable elimination can have exponential time and space complexity Constructing Bayesian Networks A Bayesian network is a correct representation of the domain only if each node is conditionally independent of its predecessors in the ordering, given its parents MarryCalls JohnCalls,Alarm,Eathquake,Bulgary)=MaryCalls Alarm) 14

15 Conditional Independence relations in Bayesian networks The topological semantics is given either of the specifications of DESCENDANTS or MARKOV BLANKET Local semantics 15

16 Example JohnCalls is indipendent of Burglary and Earthquake given the value of Alarm 16

17 Example Burglary is indipendent of JohnCalls and MaryCalls given Alarm and Earthquake Constructing Bayesian networks 1. Choose an ordering of variables X 1,,X n 2. For i = 1 to n add X i to the network select parents from X 1,,X i-1 such that P (X i Parents(X i )) = P (X i X 1,... X i-1 ) This choice of parents guarantees: P (X 1,,X n ) = π n i =1 P (X i X 1,, X i-1 ) (chain rule) = π n i =1P (X i Parents(X i )) (by construction) 17

18 The compactness of Bayesian networks is an example of locally structured systems Each subcomponent interacts directly with only bounded number of other components Constructing Bayesian networks is difficult Each variable should be directly influenced by only a few others The network topology reflects thes direct influences Example Suppose we choose the ordering M, J, A, B, E J M) = J)? 18

19 Example Suppose we choose the ordering M, J, A, B, E J M) = J)?No A J, M) = A J)? A J, M) = A)? No B A, J, M) = B A)? B A, J, M) = B)? Example Suppose we choose the ordering M, J, A, B, E J M) = J)?No A J, M) = A J)? A J, M) = A)? No B A, J, M) = B A)? Yes B A, J, M) = B)? No E B, A,J, M) = E A)? E B, A, J, M) = E A, B)? 19

20 Example Suppose we choose the ordering M, J, A, B, E J M) = J)?No A J, M) = A J)? A J, M) = A)? No B A, J, M) = B A)? Yes B A, J, M) = B)? No E B, A,J, M) = E A)? No E B, A, J, M) = E A, B)? Yes Example contd. Deciding conditional independence is hard in no causal directions (Causal models and conditional independence seem hardwired for humans!) Network is less compact: = 13 numbers needed Some links represent tenuous relationship that require difficult and unnatural probability judgment, such the probability of Earthquake given Burglary and Alarm 20

21 21

22 Aprendizagem Redes Bayesianas Como preencher as entradas numa Tabela de Probabilidade Condicional 1º Caso: Se a estrutura da rede bayesiana fôr conhecida, e todas as variavéis podem ser observadas do conjunto de treino. Então: Entrada (i,j) = yi / Pr edecessores( Yi )) utilizando os valores observados no conjunto de treino 2º Caso: Se a estrutura da rede bayesiana fôr conhecida, e algumas das variavéis não podem ser observadas no conjunto de treino. Então utiliza-se método do algoritmo do gradiente ascendente Family History Smoker Exemplo 1º caso LungCancer Emphysema Person FH S E LC PXRay D P1 Sim Sim Não Sim + Sim P2 Sim Não Não Sim - Sim P3 Sim Não Sim Não + Não P4 Não Sim Sim Sim - Sim P5 Não Sim Não Não + Não LC (FH, S) (FH, ~S)(~FH, S) (~FH, ~S) 0.5 P6 Sim Sim???? P ( yi i / Pr edecessores( Y )) = LC = Sim \ FH=Sim, S=Sim) =0.5 ~LC 22

23 Exemplo 2º caso Person FH S E LC PXRay D P1 --- Sim --- Sim + Sim P2 --- Não --- Sim - Sim P3 --- Não --- Não + Não P4 --- Sim --- Sim - Sim P5 --- Sim --- Não + Não P6 Sim Sim???? Suppose structure known, variables partially observable Similar to training neural network with hidden units In fact, can learn network conditional probability tables using gradient ascent Summary Bayesian networks provide a natural representation for (causally induced) conditional independence Topology + CPTs = compact representation of joint distribution Generally easy for domain experts to construct 23

24 24

25 25

26 -> d a,b,c)=d a,c)=0.66 d a,b,c) = "a)b)c a,b)d a,c) D a,b,c) = " < , >=< 0.66,034 > -> b a,c,d) = "# a) b)c a,b)d a,c) b a,c,d) = "a)b) c # c c a,b)d a,c) B a,c,d) = " < 0.05,0.075 >=< 0.4,0.6 > b a,c,d) = 0.6 Bayesian networks Conditional Independence Inference in Bayesian Networks Irrelevant variables Constructing Bayesian Networks Aprendizagem Redes Bayesianas Examples - Exercisos 26

27 árv dec ID3 27