Approximating discrete probability distributions with Bayesian networks

Transcription

1 Approximating dicrete probability ditribution with Bayeian network Jon Williamon Department of Philoophy King College, Str and, London, WC2R 2LS, UK Abtract I generalie the argument of [Chow & Liu 1968] to how that a Bayeian network atifying ome arbitrary contraint that bet approximate a probability ditribution i one for which mutual information weight i maximied. I give a practical procedure for finding an approximation network. The plan i firt to dicu the approximation problem and it link with Bayeian network theory. After identifying the optimal approximation, the bulk of the paper will be devoted to a propoed approximation algorithm which acrafice optimality in favour of practicality. We hall invetigate the theoretical bai and performance of thi approximation trategy. 1 THE APPROIMATION PROBLEM It can be quite unfeaible to determine, tore and manipulate a dicrete probability ditribution. Working with a ditribution over N binary variable, for intance, one mut find and tore at leat 2 N 1 probability value and it may till take too long to calculate the probability of a tatement (or event) of interet. Conequently the quetion of whether one can approximate a ditribution ufficiently accurately uing a reaonable number of pecifying value i of key importance. [Chow & Liu 1968] partially anwered thi quetion uing an innovative and highly practical method. I hall briefly outline their anwer in the language of Bayeian network theory. A Bayeian network i a directed acyclic graph (dag) over node C 1 ;:::;C N, together with a et of probability value p(c i jd i ) for each node literal c i (of the form C i v i where v i 2 fvi 1 ;:::;vki i g, the value that node C i can take) and each tate d i (a conjunction of literal c j1 ^ ::: ^ c jn ) of the direct predeceor or parent D i fc j1 ;:::;C jn g of C i. 1 Under an independence a- 1 Note that for each node the probability of one literal c can be umption, namely that any node i independent of it nondecendant conditional on it parent, a Bayeian network uffice to determine a probability ditribution, ince for each atomic Q tate of the form c 1 ^ :::^ c N we have that i p(c ijd i ) where c i i the literal and d i i the parent tate of node C i conitent with (the factor i taken to be the unconditional probability p(c i ) if C i ha no parent). Thu, depending on the connectivity in the graph, the Bayeian network repreentation of a probability function can dramatically reduce the number of probability value required to pecify the function. I hall call the number of pecifying probabilitie the complexity of a Bayeian network. While a complete dag on binary-valued node require maximum complexity 2 N 1 in a Bayeian network, a tree-haped dag only require 2N 1 pecifier, a ignificant aving. Chow and Liu howed that a tree-baed Bayeian network (with induced probability ditribution q) that bet approximate a target ditribution p (according to the cro-entropy P meaure of ditance between ditribution d(p; q) log where the are the atomic q() tate) i a maximum weight panning tree where arc between two node P are weighted by their mutual information I(C i ;C j ) c i;c j p(c i ^ c j )log p(ci^cj ) p(c i)p(c j. Chow and Liu ) alo gave an algorithm for obtaining the maximum weight panning tree: at each tage chooe the heaviet arc whoe addition enure the graph remain a foret, until one i left with a tree which we can orient a a dag. Thi propoal ha proved valuable becaue it allow the efficient approximation of a probability function. However it only partially anwer our key quetion, which aked whether one can have a ufficiently accurate approximation. The problem i that a tree-baed network may not give an approximation that i cloe enough for our purpoe, whatever they are. I hall look here at how one might bet approximate a probability ditribution by an arbitrary Bayeian network ubject to ome contraint C. Conider by way of example the determined by that of the other due to additivity I hall aume thee redundant probabilitie are left out of a pecification.

2 following contraint. ffl C 1 : no node ha more than K parent, for ome contant K. Thi bound on the number of parent erve to retrict the complexity of a Bayeian network. For intance if K 0 then the dicrete network (no arrow) i the bet approximation, if K 1then we require the bet tree approximation, and if K N 1 any complete network will determine the target ditribution exactly. It i eay to ee that if all variable are binary, the complexity of the network i le than or equal to (N K +1)2 K 1. ffl C 2 : the Bayeian network i of complexity at mot». Now if» N the only approximation i the dicrete network and if» 2 N 1 any complete network will give the target ditribution itelf. Depending on the problem in hand and available reource we will want to chooe an appropriate value for» or K which balance degree of approximation and complexity. ffl C 3 : the graph i ingly-connected. Having a ingly connected graph enure that the Bayeian network can be ued to calculate required probabilitie efficiently (in time polynomial in the number of node N). Note however that a ingly-connected network can have complexity up to 2 N 1 + N 1 on binaryvalued node, o in practice thi contraint may bet be ued with another which limit complexity. 2 THE BEST APPROIMATION One can traightforwardly generalie Chow and Liu proof that the bet tree-baed approximation i the maximum weight panning tree. The ditance between the probability function q determined by our Bayeian network and the target function p i d(p; q) log log q() log where the c i and d i are conitent with, log log N i1 N i1 N i1 log p(c i ) NY i1 p(c i jd i ) log p(c i jd i ) log p(c i ^ d i ) p(c i )p(d i ) H(p) N i1 I(C i ;D i )+ N i1 H(pj Ci ) where H(p) i the entropy of function p, I(C i ;D i ) i the mutual information between C i and it parent and H(pj Ci ) i the entropy of p retricted to node C i. The entropie are independent of the choice of Bayeian network o the ditance between the network and target ditribution i minimied jut when the total mutual information i maximied. 3 FINDING A GOOD APPROIMATION 3.1 THE IDEAL NETWORK Our contraint C come into play now, for we want to determine the maximum weight Bayeian network that atifie C: for example the maximum weight Bayeian network with node having at mot K parent. Unfortunately, it i doubtful a to whether one can efficiently find the maximum weight network which atifie arbitrary C. Clearly the brute-force way, which involve weighing all network that atify the contraint and chooing the heaviet, will be unfeaible ince the earch pace i o vat. In the abence of a practical exact method we have a choice: we can either fix C and purue pecial-cae algorithm, or we can look for general algorithm which yield network with approximately maximum weight. [Rebane & Pearl 1989] take the former approach for contraint C 3 : they give a procedure whereby if a ditribution can be repreented exactly by a ingly-connected Bayeian network then, ubject to certain aumption, that repreentation can be found. I will follow the latter coure here. 3.2 THE ADDING-ARROWS ALGORITHM Firt notice that while Chow and Liu weight were attached to arc, our mutual information weight I(C i ;D i ) (where a uual D i i the et of parent of C i ) are rather more general I hall call thee weight detached. I will however attach weight to arrow a follow: weigh arrow C j! C i in graph G by p(c i ^ c j jd i ) I(C i ;C j jd i ) p(c i ^ c j ^ d i )log p(c i jd i )p(c j jd i ) c i;c j;d i where D i i the et of C i other parent in G and the c i, c j are literal involving C i, C j and the d i are tate involving the D i. My propoed adding-arrow algorithm goe roughly a follow. At each tage chooe the heaviet arrow whoe incluion enure that C till hold and that the graph remain a dag. Stop when no more arrow can be added.

3 My uggetion i baed on the following obervation: I(A; fb; Cg) I(A; B) +I(A; CjB). PROOF: I(A; B) +I(A; CjB) 2 a;b;c a;b;c a;b;c p(a ^ b ^ c)» log p(a ^ b) p(a)p(b) + log p(a ^ cjb) p(ajb)p(cjb) p(a ^ b)p(a ^ b ^ c)p(b)p(b) p(a ^ b ^ c) log p(a)p(b)p(b)p(a ^ b)p(c ^ b) p(a ^ b ^ c) p(a ^ b ^ c) log I(A; fb; Cg) p(a)p(c ^ b) If the parent of node C are D fd 1 ;:::;D n g we can iterate the above relation to get I(C; D) I(C; D 1 )+I(C; D 2 jd 1 )+I(C; D 3 jfd 1 ;D 2 g)+::: + I(C; D n jfd 1 ;:::;D n 1 g). Thu the total detached weight of the reulting graph i jut the um of the arrow weight. The cro-entropy ditance between the network ditribution and the target ditribution i therefore minimied jut when the um of the arrow weight i maximied. The adding-arrow algorithm provide a greedy earch for the maximum total weight by maximiing each individual arrow weight. Note that the weight change at each tage o the greedy earch will not necearily find the maximum total weight: if one added a le than maximum weight arrow at ome tage, ubequent arrow may in theory have greater weight leading to a larger total. Neverthele the greedy earch i a good heuritic trategy, a we hall ee after making the algorithm more precie and after an example of the algorithm in action. In order to complete the definition of the algorithm jut note that at each tage there may be more than one maximum weight arrow conitent with C and the dag tructure. In uch a cae the previou graph will pawn everal new graph a each of thee arrow i added. Thi doe not give rie to an exploion in the number of graph under invetigation ince at each tage one can eliminate any graph that i no longer of maximum weight, and ince a the graph become more dene there are likely to be fewer optimal arrow to add. 3.3 AN EAMPLE Conider the example given by Chow and Liu (in my notation) under the contraint that the complexity of the network mut be no more than 8. Here we have four binary variable C 1 ;C 2 ;C 3 ;C 4 whoe probability ditribution can be pecified by Table 1. We tart off with a dicrete graph G 0, which induce a Bayeian network of complexity 4. Then we work out the unconditional mutual information, a Chow and Liu did, to give Table 2. Table 1: Probabilitie of atomic tate C 1 C 2 C 3 C 4 probability Table 2: Value for G 0 C 1 C 2 ; C 2 C 3 ; C 3 C 4 ;

4 Table 3: Value for G 1a C 1 C 2 ; C 1 C 3 C C 3 C 4 ; C 3 C 4 C Table 4: Value for G 1b C 1 C 2 ; C 1 C 2 C C 2 C 4 C C 3 C 4 ; Now I(C 2 ;C 3 ) i highet o we pawn two graph, G 1a with the arrow C 2! C 3 and G 1b with the arrow C 3! C 2. Each of thee graph give rie to network of complexity 5. At the next tage for G 1a we mut recalculate mutual information value involving C 3, but conditional on C 2 : I(C 1 ;C 3 jc 2 ) and I(C 3 ;C 4 jc 2 ). Note that the computation can be decompoed a follow: p(c 1 ^ c 3 jc 2 ) I(C 1 ;C 3 jc 2 ) p(c 1^c 2^c 3 )log p(c c 1 jc 2 )p(c 3 jc 2 ) 1;c 2;c 3 p(c 1 ^ c 2 ^ c 3 )p(c 2 ) p(c 1 ^ c 2 ^ c 3 )log p(c c 1 ^ c 2 )p(c 3 ^ c 2 ) 1;c 2;c 3 c 1;c 2;c 3 p(c 1 ^ c 2 ^ c 3 ) log p(c 1 ^ c 2 ^ c 3 )p(c 2 )p(c 1 )p(c 2 )p(c 3 ) p(c 1 )p(c 2 )p(c 3 )p(c 1 ^ c 2 )p(c 3 ^ c 2 ) I(C 1 ;C 2 ;C 3 ) I(C 1 ;C 2 ) I(C 2 ;C 3 ) Alternatively one might ue the identity I(C 1 ;C 3 jc 2 ) Pc 1;c 2;c 3 p(c 1 ^ c 2 ^ c 3 ) log p(c3jc1^c2) p(c 3jc 2) to calculate thee value. In Table 3 we have the value for G 1a, and the value for G 1b are in Table 4. Thu I(C 1 ;C 2 jc 3 ) ha the greatet value at thi tage. We can eliminate G 1a and add C 1! C 2 to G 1b to obtain G 2 a in Figure 1. Thi graph give rie to a network of complexity 7. We cannot next have another arrow into C 2 ince that would increae the complexity to 11. Therefore we have Table 5 for G 2 : C1 - C2 Φ* C3 ΦΦ C4 Figure 1: G 2 Table 5: Value for G 2 C 3 C 4 ; There are three contender for maximum weight and each bring the graph up to maximum complexity 8. Thu we can pawn five final graph by adding one of C 1! C 4 ;C 4! C 1 ;C 2! C 4 ;C 3! C 4 and C 4! C 3 to G 2. Each of thee graph G 3a ;:::;G 3e give rie to a cloet approximation to the target ditribution, ubject to our choen contraint. 3.4 PERFORMANCE OF THE ALGORITHM Conider the following direct generaliation of Chow and Liu original algorithm in the context of contraint C 1 :a node in the network can have at mot K parent. ffl For each node C and et D of up to K other node calculate I(C; D). Pick the C, D which maximie thi information and add an arrow from each node in D to C. Iterate thi procedure, enuring that the graph adhere to the dag tructure. Thi give another greedy earch for the optimal network and work by uing the detached weight and allowing the addition of multiple arrow at a time conequently I will call it the multiple-arrow algorithm to ditinguih it from the adding-arrow algorithm that we have been examining. Clearly thi algorithm efficiency i highly dependent on information value K, ince at the firt tage N P K i1 mut be found. If K 0we have the dicrete dag a expected. K 1 give u Chow and Liu algorithm but any larger value of K will render the method impracticable for large N. While the algorithm may not be of much ue in practice, the fact that one can add multiple arrow at once a oppoed to one arrow at a time ugget that we are weakening a contraint on our adding-arrow algorithm and therefore opening up the earch pace. Indeed, if it weren t for the added requirement of dag tructure, the multiple-arrow procedure would be guaranteed to reach the bet approximation it i only becaue adding arrow retrict the addition of further arrow that the algorithm become greedy. We hall ee now that urpriingly the N i

5 Comparion of multiple-arrow and adding- Figure 2: arrow. Figure 3: Succe under variou contraint. adding-arrow algorithm perform lightly better than the apparently more general method. In thi experiment I compared the two algorithm only for K 2 and mall value of N, due to the computational intractability of the multiple-arrow algorithm. I elected a random target ditribution by generating a Bayeian network at random and uing it to determine a ditribution. The network wa elected a follow. Firt a dag i choen at random: an ancetral order i u-randomly picked; 2 then for each node a u-random number of ucceor in the order are choen to be it children, and then thoe children are picked from the ucceor at u-random. Thu the weight i on mediumly complex graph, with very highly connected or diconnected graph le likely. 3 The pecifying probabilitie were generated uniformly at random from machine real. Having generated the network G Λ I ran the multiple-arrow algorithm (uing detached weight) to produce maximum weight graph relative to the ditribution p determined by G Λ, and then found maximum weight graph according to the adding-arrow algorithm (uing arrow weight). I then turned thee graph into Bayeian network by adding probability pecification obtained from the target network. I meaured how much further (in term of cro-entropy) the adding-arrow network wa from the target network than the multiple-arrow network. I then repeated the experiment until I could etimate the average difference in ditance for a range of value of N, the total number of (binary-valued) node. The reult are to be een in Figure 2. The reult how that the adding-arrow network are on average marginally cloer to the target ditribution than the multiple-arrow network and that the margin increae 2 I ue the expreion u-randomne to denote uniform randomne: that i, each choice in a finite partition ha the ame probability of being choen. 3 Alternatively one can generate graph a follow. For each pair of node decide whether they hould be joined by an arrow at random an arrow being a likely a none and then if there i to be an arrow decide the direction at random one direction wa a likely a the other. Reject graph that turn out not to be dag. Thu mediumly dene graph are again mot likely. It turn out that thi procedure give very imilar reulting trend. with N. Although thi i not a deciive tet becaue the multiple-arrow trategy i a much of an unknown quantity a the adding-arrow method, thi i at leat an encouraging tart. Comparing trategie for deriving Bayeian network give one teting procedure, but we can alo devie a imple tand-alone tet of the ucce of the adding-arrow approach. One can meaure the ucce of a derived network H a a percentage, 100[1 d(g Λ ;H)d(G Λ ;D)], where d meaure the cro-entropy ditance between ditribution determined by two network, G Λ i the target network and D i the dicrete network. By adding arrow one move from the dicrete network to the target network and the ucce of the derived network i the percentage of the total ditance that ha been covered. Figure 3 how the ucce of the adding-arrow algorithm for a range of node under variou contraint. A target network G Λ wa generated uing the ame random procedure a above, and derived network were generated by the greedy adding-arrow method. The experiment wa repeated o that the average ucce could be etimated. The front row how the percentage ditance gained under the contraint that the reulting graph mut be a tree or foret thi i the contraint that Chow and Liu conidered. For the econd row the contraint wa that the graph hould be ingly-connected and have no more than two parent. We can ee that the approximation are on average ignificantly cloer to the target ditribution than the foretbaed approximation. Further improvement i to be noted in the third row, where the ingle-connectedne contraint i dropped. Likewie for the fourth and the back row, where complexity i limited to 10N and 2N 2 repectively. Thank to the Machine Learning Repoitory 4 it i poible to tet the adding-arrow approach on a range of meaningful databae. However many of thee databae have miing value. While the adding-arrow approach work a normal on thee databae, the ucce meaure can be rather mileading here. The problem i that calculating cro-entropy require finding atomic tate probabilitie and that cae with miing value are ignored becaue they 4 [Blake & Merz 1998].

6 Figure 4: Databae under tructural contraint. Figure 5: Databae under complexity contraint. provide no information about thee probabilitie. However cae with miing value do provide ueful information about other probabilitie and hould not be ignored when calculating mutual information value. Conequently we are not aiming at the ditribution pecified by the databae cae with no miing value, but ome other le tangible ditribution. Furthermore, data may be miing due to ytematic bia, the knowledge of which will further alter the target ditribution. It i thi difference between the databae ditribution and the true target ditribution which make the ucce meaure (which tell u how much cloer we get to the databae ditribution) inappropriate. There i an alternative ucce meaure which i not uceptible to thi retriction, 5 but thi involve coniderably more computational time and i conidered elewhere. Conequently I focu here on databae with no miing value. Thu the adding-arrow algorithm wa run on a range of databae target ditribution and the final percentage ucce wa meaured. Figure 4 how the improvement that the relaxation of tructural contraint make on the approximation. In Figure 5 we can ee that relaxing complexity bound alo improve the approximation. 4 CONCLUSION We have een that the bet approximation to a dicrete probability ditribution by a Bayeian network ubject to an arbitrary contraint C i that which maximie detached 5 [Williamon 2000]. mutual information weight, or equivalently, conditional mutual information weight attached to arrow. While we can in theory find the bet approximation itelf, there are practical reaon for preferring a greedy adding-arrow trategy, which appear to give u cloe to the bet approximation. A to which contraint i mot appropriate will depend on the application. One may want to top adding arrow when within a certain ditance of the ditribution determined by ome databae, 6 or when a certain level of practical reliability of the approximation network i achieved: for example if the network i to be ued to olve a diagnotic problem one may want it to give the ame diagnoi a a databae ditribution in perhap 95 percent of cae, and o one may be prepared to top adding arrow when thi reliability i achieved. The fact that the adding-arrow approach doe not hinge on any particular contraint can only tand in it favour. Often in practice an expert will be able to provide qualitative caual knowledge of the domain under invetigation. While a caual dag need not atify the independence aumption itelf, it can provide ueful information about dependencie, and one can apply the method outlined here to add further arrow to the caual graph in order that the independence aumption may be more cloely atified. Thi yield a two-tage methodology for Bayeian network whereby firt one elicit a caual graph and then econdly one tranform that graph into a Bayeian network. More on two-tage Bayeian network can be found in [Williamon 2000b]. Acknowledgement Thank to the Art and Humanitie Reearch Board for funding thi reearch. Reference [Blake & Merz 1998] C.L. Blake & C.J. Merz: UCI Repoitory of machine learning databae, mlearn/mlrepoitory.html, Irvine, CA: Univerity of California, Department of Information and Computer Science. [Chow & Liu 1968] C.K. Chow & C.N. Liu: Approximating dicrete probability ditribution with dependence tree, IEEE Tranaction on Information Theory IT-14, page [Rebane & Pearl 1989] George Rebane & Judea Pearl: The recovery of caual poly-tree from tatitical 6 On the one hand one would like to cloely approximate the target ditribution, but on the other one would not want to overfit the ditribution, and one may want to acrifice ome fit in order to lower the complexity of the approximation network.

7 data, Uncertainty in Artificial Intelligence 3, page [Williamon 2000] Jon Williamon: Adding arrow to caual network, philoophy.ai report pai jw 00 b, [Williamon 2000b] Jon Williamon: Two-tage Bayeian network, philoophy.ai report pai jw 00 c,