On the Construction of the Inclusion Boundary Neighbourhood for Markov Equivalence Classes of Bayesian Network Structures

Transcription

1 On te Constrution of te Inlusion Bounry Neigouroo for Mrkov Equivlene Clsses of Byesin Network Strutures Vinent Auvry Eletril Engineering n Computer Siene Dept. University of Liège uvry@montefiore.ulg..e Astrt Te prolem of lerning Mrkov equivlene lsses of Byesin network strutures my e solve y sering for te mximum of soring metri in spe of tese lsses. Tis pper els wit te efinition n nlysis of one su ser spe. We use teoretilly motivte neigouroo, te inlusion ounry, n represent equivlene lsses y essentil grps. We sow tt tis ser spe is onnete n tt te sore of te neigours n e evlute inrementlly. We evise prtil wy of uiling tis neigouroo for n essentil grp tt is purely grpil n oes not expliitely refer to te unerlying inepenenes. We fin tt its size n e intrtle, epening on te omplexity of te essentil grp of te equivlene lss. Te empsis is put on te potentil use of tis spe wit greey illliming ser. 1 INTRODUCTION Lerning Byesin network strutures is often formulte s isrete optimiztion prolem : te ser for n yli struture mximizing given soring metri. If we o not give ny usl semntis to te rrow, we my onsier tt two Byesin network strutures re (istriution) equivlent if tey n e use to represent te sme set of proility istriutions. Moreover, ommon soring metris ssign te sme vlue to equivlent strutures n re tus lso si equivlent. Te ignorne of tese fts my egre te performne of greey lerning lgoritms, ut tken into ount, tey n lso improve it (see [Cikering, 2002] n [Anersson et l., 1999]). In orer to o so, we my ssign to e equivlene lss te sore of its elements n ser for te est lss. Te efinition of ser spe of equivlene lsses is not Louis Weenkel Eletril Engineering n Computer Siene Dept. University of Liège Louis.Weenkel@ulg..e trivil n is te topi of tis pper. Our spe is rterize y te ft tt te lsses re represente y essentil grps n te use of notion of neigouroo lrey propose for Byesin network strutures : te inlusion ounry neigouroo. Altoug tis new spe n e use y lerning lgoritms wit vrious ser strtegies, we put te empsis on greey ill-liming ser. As we will see, te size of tis neigouroo is very lrge in te worst se ut our results n serve s sis for sensile pproximtions n re interesting y temselves. In setion 2, we present mteril use susequently n mostly relevnt to equivlene issues. Te inlusion ounry is formlly efine in setion 3. Setion 4 overs te genertion of te neigouroo n ow to sore its elements inrementlly. Setion 5 gives erly omments on te ypotetil pplition of te spe wit greey ill-liming ser. We onlue in setion 6. As reviewer pointe out, te reent pper [Cikering, 2002] prtly els wit te sme topi n presents, y noter ppro, orroorting results. 2 PRELIMINARIES Tis setion reviews notions require for preise unerstning of te pper, settles our nottions n presents some of our teorems. Te reing of tese ltter teorems is rter teious n n e elye until tey re tully use in setion GRAPHICAL NOTIONS A grp G is pir G = (V G, E G ), were V G is finite set of verties n E G is suset of (V G V G ) \ {(, ) V G }. E G efines te struture of G in te following wy : G ontins line etween n if (, ) E G n (, ) E G, wi is note G, G ontins n rrow from to if (, ) E G n (, ) E G, wi is note G,

2 G ontins n ege etween n if, or G, wi is note G. A grp is omplete if tere is n ege etween every pir of istint verties. In tis pper, ll grps onsiere in te ser spe ve te sme set of verties V. Tere is tus one-to-one orresponene etween grps n strutures. Te set of prents p G (x) of vertex x in grp G onsists of te verties y su tt y x G. A sugrp G of grp G is grp su tt 1 V G V G n E G E G (V G V G ). Te inue sugrp G A, were A V G is te sugrp of G su tt V GA = A n E GA = E G (A A). A set of verties is omplete in G if te sugrp of G it inues is omplete. A pt is sequene x 0,..., x n of istint verties were (x i, x i+1 ) E G, i = 0,..., n 1. Tis pt is unirete if x n,..., x 0 is lso pt. Te reltion is n equivlene reltion etween verties efine y = or tere exists n unirete pt etween n. Tis reltion prtitions te set of verties of grp into equivlene lsses. A yle is pt wit te moifition tt x 0 = x n. A yle is irete if x i x i+1 E G for lest one i {0,..., n 1}. A v-struture (, {t 1, t 2 }) of G, were, t 1 n t 2 re istint verties is pir su tt t 1 G, t 2 G n t 1 t 2 G. V (G) enotes te set of v-strutures of G. An rrow p q G is protete in G if p G (p) p G (q) \ {p}. An rrow p q G is strongly protete if G inues t lest one of te sugrps of figure 1. p () q p () q p () q p () Figure 1: Strongly Protete Arrow p q q Aoring to teir properties, grps my e lssifie in irete, unirete n in grps. A irete grp D = (V D, E D ) is grp witout ny line. An yli irete grp or DAG is irete grp tt ontins no (irete) yle. Te set of ll DAGs efine wit te sme set of verties V is note D. Oviously, n rrow D is protete if D inues t lest one of te sugrps (), () or () of figure 1. An unirete grp U = (V U, E U ) is grp witout ny rrow. Te skeleton S(G) of grp G is te unirete 1 Trougout tis pper, A B mens tt A is proper suset of B, wile A B mens tt A = B or A B. grp resulting from ignoring te orienttion of te rrows in G. An unirete grp is orl if every (unirete) yle of lengt 4 s or, i.e. line etween two non-onseutive verties of te yle. A perfet orering of n unirete grp U is totl orering of V U su tt te irete grp D otine y ireting every line U from to if preees in te orering is yli n ontins no v-struture. D is lle perfet irete version of U. Te following teorem ols true (see e.g. [Cowell et l., 1999]). Teorem 2.1 An unirete grp is orl if, n only if, it mits t lest one perfet orering. Mximum rinlity ser (MCS) is n lgoritm tt eks if n unirete grp is orl n, if so, provies perfet orering. A esription of MCS n e foun in [Cowell et l., 1999]. Let us just note tt MCS n e use to immeitely n onstrutively prove lemm 2.2. Lemm 2.2 Given orl unirete grp U n non-empty set of verties A V U inuing omplete sugrp, ny permuttion of A is te eginning of perfet orering of U. For exmple, let n e pir of jent verties of U. We eue from te pplition of te lemm 2.2 wit A = {} tt tere exists perfet irete version D of U su tt D. Te next lemm is use in teorem 2.7. Lemm 2.3 Te removl of from orl unirete grp U tt oes not inue te sugrp of figure 9 for ny V U proues orl unirete grp. Proof. Let us prove tt if is or for yle of lengt m 4, ten U inues te sugrp of figure 9. - ivies tt yle into two su-yles. Let x e one of tem. On te one n, te existene of yle of lengt 3 ontining implies tt U inues te sugrp of figure 9. On te oter n, if tere exists yle of lengt n 4 ontining ten tere exists yle ontining n of lengt n su tt 3 n n 1. Su n e otine s follows. By te orlity of U, s (t lest) one or. Tt or ivies into two su-yles. n e osen s te su-yle ontining. By n inutive resoning strting wit = x n ening wit n = 3, we onlue tt U inues te sugrp of figure 9. A onsistent extension of grp G is DAG D su tt D s te sme skeleton s G, te sme set of v-strutures n every rrow of G is present in D. Dor n Trsi (see [Dor n Trsi, 1992]) foun n lgoritm to ek if grp possesses onsistent extension n, if so, to fin one.

3 A in grp C is grp witout ny irete yle. DAGs n unirete grps re speil ses of in grps. Te set of in omponents of in grp C is te set of equivlene lsses of verties inue y te reltion in C. E sugrp of C inue y in omponent is unirete, euse oterwise C woul ontin irete yle. 2.2 BAYESIAN NETWORKS Grps re sometimes use to represent sets of onitionl inepenenes etween rnom vriles. A Byesin network B for set of rnom vriles X = {x 1,..., x n } is pir (D, Θ), were D is DAG efine on set of verties in one-to-one orresponene wit X n Θ = {θ 1,..., θ n } is set of prmeters su tt e θ i efines onitionl proility istriution P (x i p D (x i )). Su Byesin network represents te proility istriution P (X) efine s P (X) = n i=1 P (x i p D (x i )). Let us efine I(D) s te set of onitionl inepenenes U V W 2 su tt W -seprtes 3 U n V in D. One n sow tt te inepenenes of I(D) re verifie in P (X). Conversely, if proility istriution P (X) verifies te inepenenes of set I(D), ten P (X) n e eompose in prout n i=1 P (x i p D (x i )) n is tus representle y Byesin network efine on D. 2.3 EQUIVALENCE OF DAGS Two DAGs K, L D (or teir struture) re inepenene (or Mrkov) equivlent if I(K) = I(L). Tis reltion inues equivlene lsses in D. Distriution equivlene implies inepenene equivlene, ut te onverse is not true in generl. Te susequent evelopments re ll se on inepenene equivlene, even if not expliitely mentione. To use tem in our lerning prolem, we ple tis pper in ny ontext were inepenene n istriution equivlenes re logilly equivlent. Verm n Perl (1990) erive te following teorem. Teorem 2.4 Two DAGs re equivlent if, n only if, tey ve te sme skeleton n te sme set of v-strutures. An equivlene lss is tus rterize y skeleton n set of v-strutures. Te essentil grp 4 E = (V E, E E ) of n equivlene 2 Te nottion U V W mens tt te sets of vriles U n V re inepenent given te set of vriles W. 3 See [Cowell et l., 1999] or [Perl, 1988] for efinition of -seprtion n ll te etils. 4 Essentil grps re lso lle omplete prtilly irete yli grps. lss note [E] is efine y V E = V D for ny D [E] n E E = D [E] E D. Let us mke few omments out E. E s te sme skeleton s te DAGs of [E]. E s te rrow if, n only if, every DAG of [E] lso s it. Similrly, E if, n only if, tere exist two DAGs of [E] su tt one s te rrow n te oter s. For exmple, we eue from teorem 2.4 tt E n te DAGs of [E] ve te sme skeleton n set of v-strutures. Let D enote te essentil grp of te equivlene lss ontining te DAG D. Te following teorem ensures tt te essentil grp E n e use s representtion of [E] (see [Anersson et l., 1999]). Teorem 2.5 Let K, L e two DAGs. I(K) = I(L) if, n only if, K = L. Te set of (onitionl) inepenenes I(E) represente y n essentil grp E is efine s te set I(D) of (onitionl) inepenenes of ny D [E]. Essentil grps re rterize y teorem of Anersson (see [Anersson et l., 1999]). Teorem 2.6 A grp G is n essentil grp, i.e. G = D for some DAG D if, n only if, G stisfies te following onitions : G is in grp ; for every in omponent τ of G, G τ is orl ; G oes not inue te sugrp ; every rrow of G is strongly protete in G. Let E enote te set of essentil grps efine on te set of verties V. We prove te following teorem, use in setion 4.2. Teorem 2.7 Let E e n essentil grp su tt E n E oes not inue te sugrp of figure 9 for ny V E. Te grp G otine y removing from E is essentil. Proof. Oviously, G is in grp n oes not inue v 1 v 2 v 3. Let S e n inue sugrp of E strongly proteting n rrow p q. If S is of te type of figure 1() n S, ten G inues sugrp S of te type of figure 1(). Oterwise, S is lso n inue sugrp of G. Every rrow of G is tus strongly protete in G. Let τ e in omponent of E. If E τ ten y lemm 2.3, G τ is orl. Oterwise, G τ = E τ. Hene, every sugrp of G inue y one of its in omponents is orl. By teorem 2.6, G is n essentil grp.

4 Te essentil grp D n e otine from D y te following lgoritm n teorem from [Anersson et l., 1999]. Algoritm 2.1 Let G 0 e grp. For i 1, onvert every rrow G i 1 tt is not strongly protete in G i 1 into line, otining grp G i. Stop s soon s G k = G k+1 (k 0) n return G k. on {,, }. Similrly, in se 2(f), is strongly protete. By onstrution, S tus inues one of te sugrps of figure 2, n in prtiulr S. Suppose now tt Q(L). By ypotesis (i), is strongly protete in L y (t lest) one inue sugrp of te type of figure 1(), 1() or 1(). Beuse every rrow of L is present in S, is lso strongly protete in S n tus S. Teorem 2.8 If G 0 = D, ten lgoritm 2.1 returns D. Note tt oter lgoritms exist, ut we exten tis one y teorem Let us first introue lemm n some nottion. Define Q(G) s te set of rrows of te grp G tt re strongly protete in G only y one or more sugrps of te type of figure 1(). Lemm 2.9 Let S n L e grps su tt (i) every rrow of L is strongly protete in L, (ii) every rrow of L is present in S, (iii) for e Q(L), S inues one of te sugrps of figure 2. Teorem 2.10 Let D e DAG n G 0 grp su tt (i) V G0 = V D, (ii) every rrow of G 0 is present in D, (iii) every rrow of D is present in G 0, (iv) for e Q(D ), G 0 oes not inue te sugrp of figure 3 for ny, V G0. Figure 3: Forien Sugrp () () () (e) () (f) Figure 2: Inue Sugrps Let S e te grp otine from S y onverting every non strongly protete rrow into line. Te grp S stisfies te ove ypoteses onerning S. Proof. Let e n rrow of L. Suppose first tt Q(L). Let us onsier te inue sugrps of figure 2. In e se, te rrows n strongly protet one noter y sugrp of te type of figure 1() inue on {,, }. In se 2(), is strongly protete y n inue sugrp of te type of figure 1(), wile in te oter ses, tt rrow is strongly protete y (t lest) one inue sugrp of te type of figure 1(). Moreover, in se 2(e), is strongly protete y sugrp of te type of figure 1() Algoritm 2.1 pplie to G 0 returns D. Proof. Let G 0,..., G k e te sequene of grps proue y lgoritm 2.1. Oviously, S(G k ) = S(D ). Let us sow tt G k n D ve te sme rrows n tus G k = D. On te one n, let e n rrow of Q(D ). D tus inues sugrp of te type of figure 1() to strongly protet. By ypoteses (iii) n (iv), G 0 inues one of te sugrps of figure 2. By n inutive pplition of lemm 2.9 eginning wit S = G 0 n L = D, we eue tt te rrows of D re present in G k. On te oter n, note tt y ypotesis (ii) n te esription of lgoritm 2.1, every rrow of G k is present in D. If Q(G k ), D tus inues sugrp of te type of figure 2(), 2(e) or 2(f). We eue from teorem 2.8 n n inutive pplition of lemm 2.9 eginning wit S = D n L = G k tt te rrows of G k re present in D. Te first tree ypoteses of tis teorem re, for exmple, stisfie if G 0 is otine from D y onverting some rrows D su tt D into lines, or if G 0 is su tt V G0 = V D n E G0 = G X E G, were X [D ]. Conversely, every DAG D [E] n e reovere from E y teorem 2.11 (see [Anersson et l., 1999]).

5 Teorem 2.11 D [E] if, n only if, D is otine from E y orienting te lines of every unirete (orl) sugrp inue y in omponent of E oring to perfet orering. As expete, y efinition of te perfetion of n orering, te orienttion of te lines oes not introue ny new v- struture. Besies, one n see tt te elements of [E] re te onsistent extensions of E. Dor n Trsi s lgoritm pplie to E tus returns D [E]. 2.4 SCORING METRICS A soring metri sore for DAGs is eomposle if it n e written s sum (or prout) of funtions 5 of only one vertex n its prents, i.e. sore(d) = x V f(x, p D (x)) A soring metri for DAGs is equivlent if it ssigns te sme vlue to equivlent DAGs. In tis pper, tis property is suppose to ol, s for exmple wit te well-known BDe sore. In su se, te sore of n equivlene lss (or its essentil grp) is efine s te sore of (ny of) its elements. 3 DEFINITION OF THE INCLUSION BOUNDARY NEIGHBOURHOOD Te neigouroo of Byesin network struture is often efine in terms of opertions performe on tt struture, su s te ition, removl or reversl of n rrow. For exmple, te grps of figure 4 re typilly neigours. Tis kin of neigouroo is onstrute very sim () () Figure 4: Ajent DAGs ply n effiiently. Furtermore, if eomposle soring metri is use, te sore of te neigours of struture n e lulte inrementlly, i.e. wit just few evlutions of f. Te sme ie is pplile to ser spes of essentil grps, wit opertors su s te ition of n rrow, line or v-struture, te reversl of n rrow,... However, te sitution is omplite y te onstrints on essentil grps : te grp moifie y n opertor must 5 Te epenene of te metri on te t is not me expliit. stisfy te onitions of teorem 2.6. Te reent pper [Cikering, 2002] sows tt tese prolems n e overome y refully oosing te opertors so s to finlly get n effiient lgoritm, n in prtiulr keep n inrementl evlution of te soring metri. In tese ltter two ses, te neigouroo is efine y moifitions performe on te grp, witout ny referene to te inepenenes represente. Inste, it my e efine s its inlusion ounry. Let G e set of grps representing (onitionl) inepenenes. A grp G G elongs to te inlusion ounry wit respet to G of G G if G G n one of te following mutully exlusive onitions is stisfie : (i) I(G ) = I(G), (ii) I(G) I(G ) n tere is no G G verifying I(G) I(G ) I(G ), (iii) I(G ) I(G) n tere is no G G verifying I(G ) I(G ) I(G). Tis ie s lrey een use in [Kočk n Cstelo, 2001] wit G = D, i.e. wit Byesin network strutures. Te DAGs of figure 4 re ten no longer neigours. We trnspose tis ie to efine spe se on equivlene lsses represente y essentil grps, i.e. G = E. In tis se, te first onition is never stisfie. For prtiulr E E te set of essentil grps efine y (ii) n (iii) re respetively note N + (E) n N (E). By efinition, tese sets never interset. Note tt if M N + (N), ten oviously N N (M), n onversely. Our ser spe is onnete if, etween ny M, N E, tere exists finite sequene of essentil grps E 1,..., E l, su tt E 1 = M, E l = N n E i+1 is neigour of E i for i = 1,..., l 1. Tis property is importnt for lol ser. Teorem 3.1 Te ser spe is onnete. Proof. Tere exists n essentil grp U efine on te finite set of verties V = {v 1,..., v n } (i.e. U E) n su tt I(U) =. Inee, let D e te DAG su tt p D (v i ) = {v 1,..., v i 1 }. We ve U = D. For e E E, te following fts ol. If E U, I(U) I(E) n tus N (E). For ll G N (E), I(G) < I(E) 6. Te set I(E) is finite. Hene, tere exists finite sequene of essentil grps E 1,..., E l, su tt E 1 = E, E l = U n E i+1 N (E i ). By te symmetry of te neigouroo, te sequene E l,..., E 1 is su tt E i N + (E i+1 ). For ll M, N E, tere tus exists sequene M,..., U,..., N wit te require properties. 6 X enotes te rinlity of te set X.

6 Some questions still nee n nswer. Is te size of tis neigouroo trtle? Cn te elements of te neigouroo e generte effiiently n/or sore inrementlly? Te next setion resses tem y expliitely uiling N + (E) n N (E). 4 CONSTRUCTION OF THE INCLUSION BOUNDARY NEIGHBOURHOOD Te onstrution of N(E) = N + (E) N (E) from te onitions (ii) n (iii) is not immeite. Moreover, given G E, it is not trivil to ek weter G N(E) or not. Tese iffiulties stem from te ft tt te onitions re expresse troug I(E) inste of E s grpil omponents. Te following lemm 7 simplifies te expression of te neigouroo. Lemm 4.1 N + (E) = {G E K, L D : K = G, L = E n K is otine from L y te removl of one rrow}, N (E) = {G E K, L D : K = G, L = E n K is otine from L y te ition of one rrow}. If eomposle soring metri is use, n importnt orollry is te possiility to evlute inrementlly te sore of E s neigours from te sore of E. Using te nottions of te lst lemm, for e G N(E) we ve G sore = sore(g) sore(e), = sore(k) sore(l), = f(x, p K (x)) f(x, p L (x)), (1) were x is te estintion of te rrow e or remove in L. We see tt very little is suffiient to estimte te inrement in sore, in prtiulr te omplete knowlege of G is not neee. Te expressions of lemm 4.1 suggest prtil wy of uiling N(E). Given DAG D, let R(D) e te set of DAGs tt n e onstrute y removing or ing n rrow to D. Oviously, we ve N(E) = D [E] M R(D) {M }. However, tis ppro is reunnt in te sense tt te sets of te unions re not neessrily isjoint. For exmple, te DAGs of figures 5() n 5() elong to te equivlene lss represente y te essentil grp of figure 5(). Te removl of from ot of tese proues DAGs of te sme lss, represente y figure 5(). 7 Our proof of tis lemm uses Meek s onjeture, reently prove in [Cikering, 2002], n n e otine upon request to te first utor. () () () () Figure 5: Removl Prouing Equivlent DAGs To irumvent tis pitfll, we ivie our tsk in two prts : te ientifition of te neigours n, if neessry, teir onstrution. Let : N(E) X e n injetive funtion, i.e. su tt G 1 G 2 implies (G 1 ) (G 2 ). In tis pper, is lle rteriztion funtion n (G) rteriztion of G. A given x X is vli if it rterizes G N(E), i.e. tere exists G N(E) su tt (G) = x. Wit rteriztion funtion, we my uil N(E) y first ientifying te vli elements of X n ten, for e su element, otining te orresponing G N(E). We will use two su funtions : 1 to uil N (E) n 2 for N + (E). Tey re efine s follows. We eue from teorem 2.4 tt e G N(E) is rterize y its skeleton S(G) n set of v-strutures V (G). Let T e te set of skeletons tt re otine from S(E) y removing or ing line. Tere is one-to-one orresponene etween T n te set of unorere pirs of verties su tt e t T is ssoite to te pir {, } of verties tt re te enpoints of te line remove or e to otin t from E. We see from lemm 4.1 tt, for e G N(E), S(G) T. Besies, te set V (G) for G N(E) n e eompose s (V (G) \ V (E)) (V (E) \ (V (E) \ V (G))). By lemm 4.1, V (G) \ V (E) n V (E) \ V (G) re te sets of v-strutures respetively rete n estroye y te ition or removl 8 of n rrow in D [E]. We eue from te following ovious lemm tt V (E) \ V (G) epens only on S(G) (or te pir of verties orresponing to it). Lemm 4.2 If te exeution of te opertion orresponing to given pir {, } estroys v-struture v from D [E] ten tt opertion estroys v from every G [E]. Gtering tese oservtions, we ve first rteriztion funtion 1 : N(E) X 1 : G 1 (G) = (g 1 (G), g 2 (G)) = ({, }, O), were {, } re te verties ssoite to S(G) n O = V (G) \ V (E). Te previous isussion lso les to te vliity onition of lemm 4.3. Lemm 4.3 A pir ({, }, O) X 1 rterizes G N(E) if, n only if, K, L D : K = G, L = E, K is 8 Tis opertion on DAGs mirrors te opertion performe on S(E) to otin S(G).

7 otine from L y performing te opertion ssoite to {, } n O = V (K) \ V (L). Let A(v, {, }) [E] enote te set of DAGs were te opertion ssoite to {, } retes te v-struture v n let R({, }) enote te set {v A(v, {, }) }. Oviously, if ({, }, O) is vli ten O R({, }). We lso use sligtly moifie version of 1, efine s follows. Given {, }, let Y e te set O Z O were Z = {O ({, }, O) X 1 is vli}. In oter wors, Y onsists of te v-strutures tt re rete in every DAG of [E] y te opertion. Te funtion 2 : N(E) X 2 : G 2 (G) = (g 1 (G), g 2 (G) \ Y ) is injetive. Let W ({, }) enote te set {v A(v, {, }) n A(v, {, }) [E]}. Te vliity onition for 2 is given y te next lemm. Lemm 4.4 A pir ({, }, O) X 2 rterizes G N(E) if, n only if, O W ({, }) n K, L D : K = G, L = E, K is otine from L y performing te opertion ssoite to {, }, O V (K) \ V (L) n W ({, }) \ O V (K) \ V (L). In setions 4.1 to 4.3 we present our meto to ientify te vli rteriztions n etermine te orresponing neigours n inrements in sore. As we will see, te meto iffers if tere is n rrow etween n in E, E or E. Let N (E) e te suset of N(E) su tt its elements ve te sme skeleton, rterize y {, }. By nlogy wit te oter type of neigouroo ite in setion 3, we efine tree pseuoopertors 9 : removl of E, removl of E n ition of n ege etween n to E use in te orresponing situtions n returning N (E). Te onstrution of N(E) n ten proee y enumerting te unorere pirs of verties n, for e, lling te orresponing pseuo-opertor. We ve te following teorem. 4.1 REMOVAL OF AN ARROW E In te ontext of pplition of tis pseuo-opertor, {, } is fixe n te ssoite opertion is te removl of. We use te rteriztion funtion 2. Te lemm 4.6 oviously ols true. Lemm 4.6 Te removl of n rrow from DAG retes te v-struture (, {t 1, t 2 }) if, n only if, {t 1, t 2 } = {, } n te DAG inues te sugrp of figure 6. Figure 6: Cretion of V-Struture Te set W ({, }) is esily ientifie grpilly from E wit te following teorem. Teorem 4.7 (, {t 1, t 2 }) W 10 if, n only if, {t 1, t 2 } = {, } n E inues te sugrp of figure 7. Figure 7: Inue Sugrp of E Proof. If we remin te mening of line of n essentil grp, te suffiient prt is trivil 11. By lemm 4.6 n te efinition of W, {t 1, t 2 } = {, } n tere exists D [E] inuing te sugrp of figure 6. Tere lso exists K [E] were te removl of oes not rete (, {, }). Su K must ve n te sme skeleton s D. It tus inues te yli sugrp 8() or 8(). By te yliity of E, K must inue 8() n Teorem 4.5 For e {, }, N (E) is non-empty. Proof. Let D e DAG of [E]. If D, ten te grp D otine y removing tt ege from D is oviously DAG. If D ten tere exists DAG D otine from D y ing n rrow etween n. Inee, suppose tt te ition of retes te yle,, v i1,..., v ik, n te ition of retes te yle,, v ik+1,..., v ik+l,. D woul possess te yle, v i1,..., v ik,, v ik+1,..., v ik+l, n woul not e DAG. By lemm 4.1, D N (E). 9 Tese re not opertors in te usul sense euse tey return set of sttes inste of single one. () () Figure 8: Inue Sugrps of K E. E tus inues te sugrp of figure 7. Te vli rteriztions O re oviously susets of W n n e otine wit te following teorem. 10 Te epenene on {, } is me impliit for revity. 11 Te suffiient prt is not use in teorem 4.8.

8 Teorem 4.8 O is vli if, n only if, O W n te set C = { (, {, }) W \ O} is omplete in E. Proof. By lemm 4.4, O W is vli if, n only if, tere exists D [E] su tt D for { (, {, }) O} n D for C. Te existene of su D is eke wit teorem Let τ e te in omponent of E ontining { (, {, }) W }. Te onstrints on te orienttion of te lines of E to otin D re only relte to E τ. E sugrp of E inue y noter in omponent n tus e irete oring to perfet orering 12 inepenently. Hene, tere exists su D if, n only if, tere exists perfet orering of E τ leing to te require rrows. On te one n, let o e su perfet orering. Te perfet irete version H of E τ s no v-struture n te rrows for C. Any verties i, j C must e jent in H, euse oterwise H woul possess te v-struture (, { i, j }). We tus ve i j E. On te oter n, suppose tt C is omplete. C {} is ten lso omplete. By lemm 2.2, for ny permuttion 1,..., k of C, 1,..., k, is te eginning of perfet orering o. Su n orering les to te require rrows. Tis teorem s n immeite orollry. Corollry 4.9 Tere is one-to-one mpping etween N (E) n te omplete susets of { (, {, }) W }. Suppose tt O rterizes E N (E). Let us isuss te onstrution of E. We use te nottions of te previous teorem. Let D e DAG otine from E y (i) removing, (ii) ireting te lines of E τ oring to o n (iii), for e sugrp E α of E inue y noter in omponent, ireting its lines oring to perfet orering. From te proof of teorem 4.8, we see tt te set B of tese DAGs is suset of [E ]. Te grp G su tt E G = D B E D n lerly e onstrute from E y performing te steps (i) n (ii). Moreover, y symmetry of o, we know tt E C is unirete. Let us uniret te rrows of G tt re present in G C. We ve te following result. Teorem 4.10 Algoritm 2.1 pplie to G returns E. Proof. Let us sow tt G stisfies te ypoteses of teorem Oviously, G stisfies te first tree onitions. Let us sow tt G oes not inue p q r n tus stisfies te lst ypotesis. Suppose tt G inues p q- r. If p q E, ten y onstrution of G, E lso inues p q r, wi is impossile sine E is n essentil grp. Oterwise, using te nottions of teorem 4.8, q, r C n p τ \ C. But te rrows of G re irete oring to n orering eginning wit permuttion of C. Tus, tere n not e n rrow from vertex of τ \ C 12 Te existene of su n orering is gurntee y teorems 2.6 n 2.1. to one of C. In sense, E n tus e onstrute inrementlly from E. Te inrement in sore is esily evlute wit formul (1), yieling : G sore = f(, (p E () \ {}) C) f(, p E () C) 4.2 REMOVAL OF A LINE E Te opertion ssoite to {, } is te removl of te rrow etween n. We use 2. Te set W ({, }) n e ientifie grpilly y te following teorem, te proof of wi is very similr to tt of teorem 4.7. Teorem 4.11 (, {t 1, t 2 }) W if, n only if, {t 1, t 2 } = {, } n E inues te sugrp of figure 9. Figure 9: Inue Sugrps of E Te vli rteriztions O re susets of W n re foun wit teorem 4.12, wose terms re ientil to tose of teorem 4.8. Teorem 4.12 O is vli if, n only if, O W n te set C = { (, {, }) W \ O} is omplete in E. Proof. Let τ e te in omponent of E ontining { (, {, }) W }. One gin, O W is vli if, n only if, tere exists perfet orering o of E τ su tt te removl from te perfet irete version H of E τ retes te v-strutures of O ut not tose of W \ O. One te one n, if C is omplete, ten C {, } is omplete. By lemm 2.2, if 1,..., k is permuttion of C ten tere exists perfet orering o eginning wit 1,..., k,,. Tt o s te require properties. On te oter n, let o e su perfet orering. Suppose tt H 13. For e C, H {,,} is te sugrp of figure 10() or 10(). As n e seen, y omining () () Figure 10: Inue Sugrps of E tose sugrps, every i, j C must e jent in H. 13 Tis is mtter of nottion.

9 Oterwise H woul possess te v-struture (, { i, j }). C is tus omplete in E. Given O rterizing E N (E), E n e onstrute wit proeure nlogous to te one given in setion 4.1. Let G e te grp otine from E y removing n ireting te lines of E τ oring to perfet orering of E τ eginning wit permuttion of C followe y,. Te rrows of G present in G C re ten unirete. One n see from te proof of teorem 4.12 tt G stisfies te ypoteses of teorem 2.10 n n tus e use s strting point for lgoritm 2.1. Besies, if E oes not inue sugrp of te type of figure 9, i.e. W =, ten E n e onstrute y simpler proeure. Inee, y teorem 2.7, te grp G otine y removing from E is essentil. Moreover, G s te sme skeleton n set of v-strutures s E. Hene, E = G. Te inrement in sore is given y te next formul, were n n e permute y symmetry. G sore = f(, p E () C) f(, p E () C {}) 4.3 ADDITION OF AN EDGE TO E Te opertion ssoite to {, } is te ition of n rrow etween n. We use 1. We ve te following lemm. Lemm 4.13 If (, {t 1, t 2 }) R({, }), ten (i) {t 1, t 2 } = {, t}, = n E inues te sugrp of figure 11() or 11(), or (ii) {t 1, t 2 } = {, t}, = n E inues te sugrp of figure 11() or 11(). t () t () t () Figure 11: Inue Sugrps of E Let P e te set of v-strutures verifying te tesis of lemm For e vli ({, }, O), we ve O R({, }) P. We in t fin simple grpil neessry n suffiient onstrints on E to etermine te vliity of given rteriztion O, ut we ve lemm 4.14 n teorem Let us introue some nottion. Let P i, i = 1,..., 4 e te prtition of P su tt, for e element of P 1, P 2, P 3 or P 4, E inues sugrp of te type of, respetively, figure 11(), 11(), 11() or 11(). E vli O P n e eompose into te sets O i = O P i, i = 1,..., 4. t () Lemm 4.14 If O is vli, ten O P n (t lest) one of te two following onitions is stisfie. (i) O 2 = P 2, O 3 = O 4 = n F 1 = {t (, {t, }) O 1 } is omplete in E ; (ii) O 4 = P 4, O 1 = O 2 = n F 3 = {t (, {t, }) O 3 } is omplete in E. Proof. By lemm 4.3, K, L D : L = E, K is otine y ing to L n rrow etween n, n O = V (K) \ V (L). Let τ e te in omponent of E ontining te set of verties {t (, {t, }) P 1 }. By teorem 2.11, te rrows of L τ re oriente oring to perfet orering of E τ. Moreover, t L τ for t F 1. Every t i, t j re jent in L τ, euse oterwise L τ woul possess te v-struture (, {t i, t j }). F 1 is tus omplete in E. Similrly, we eue tt F 3 is omplete in E. If K, ten, y lemm 4.6, O 2 = P 2 n O 3 = O 4 =. If K, ten O 4 = P 4 n O 1 = O 2 =. Suppose tt given O stisfies tese onitions. Let G(O) e te grp otine from E s follows. If O =, simply. Oterwise 14, if (i) is stisfie, n iret every line t su tt t F 1 towrs, wile if (ii) is stisfie, n iret every line t su tt t F 3 towrs. We n ek te vliity of O wit te next teorem n Dor n Trsi s lgoritm. Teorem 4.15 O is vli if, n only if, O stisfies te onitions of lemm 4.14 n G(O) s onsistent extension. Proof. Suppose G(O) s onsistent extension M. Te essentil grp M is rterize y ({, }, O). Inee, S(M ) = S(G) n V (M ) \ V (E) = V (M) \ V (E) = V (G) \ V (E). By onstrution, S(G) is rterize y {, } n V (G) \ V (E) = O. Suppose tt O rterizes E N (E). Let K e one DAG wose existene is mentione in lemm 4.3. K is onsistent extension of G(O). As tis proof sows, given O rterizing E N (E), E n e otine y pplying lgoritm 2.1 to onsistent extension M of G(O) 15. M n lso e use to evlute te inrement in sore. 5 APPLICATION TO LEARNING In tis setion, te ypotetil use of our ser spe wit greey ill-liming is isusse. Tis spe s vlule 14 Te onitions (i) n (ii) of teorem 4.14 re now exlusive. 15 Te glol nture of te yliity onstrint prevents te inrementl onstrution of te essentil grps wit te previous proeure.

10 properties. First, it is onnete. Moreover, te sore of e neigour E of E n e evlute inrementlly from E s sore n witout onstruting E. If we o nee E n E N + (E), ten it n e uilt from E inrementlly y retining priori some of its lines. Te min rwk of tis ser spe is tt te size of te neigouroo n e intrtle for struturlly omplex essentil grps. Inee, let e te numer of verties of te lrgest omplete unirete inue sugrp of E. Setions 4.1 to 4.3 tell us tt, in te worst se, te numer of elements of N(E) is exponentil in. Let us mke some erly omments on te impt of tis size on two opposite wys of strting greey ill-liming ser. Suppose tt te ser strts wit te empty essentil grp n ten s eges. We expet tt te men size of N(E) n tus te omputtionl ost will ugment s we progress in te spe. Tis eviour is ertinly prolemti, ut proly omes wit growing nee for more t to support te suessive removl of te inepenenes. Suppose now tt te ser strts wit te omplete essentil grp n ten prunes it. In tt se, our neigouroo is lerly inpproprite. Tis n e interprete s te ft tt it is too fine-grine for pruning, t lest in te erly steps, n tt more ggressive strtegy soul e use. 6 CONCLUSION Te topi of tis pper is te onstrution n nlysis of ser spe of Mrkov equivlene lsses of Byesin networks represente y essentil grps n wit te inlusion ounry neigouroo. Our nlysis sows tt tis spe is onnete n te sore of e neigour of n equivlene lss n e evlute inrementlly from te sore of tt lss. Anoter importnt ontriution is te suggestion of proeure to tully uil te neigouroo of lss. As yprout, oun on te size of te neigouroo tt n e lulte very simply priori is etermine. Tis work n e extene y reful estimtion of te impt of tt size on te lerning lgoritms to possily propose pproximtions. In next step, tis spe n e ompre to oters, se on Byesin networks or equivlene lsses, for exmple on te sis of te performne of te lgoritms using tem. Tenil report, Deprtment of Sttistis, University of Wsington, [Cikering, 1996] Dvi Mxwell Cikering. Lerning equivlene lsses of Byesin network strutures. In E. Horvitz n F. Jensen, eitors, Proeeings of Twelft Conferene on Unertinty in Artifiil Intelligene, pges Morgn Kufmnn, August [Cikering, 2002] Dvi Mxwell Cikering. Lerning equivlene lsses of Byesin-network strutures. Journl of Mine Lerning Reser, 2: , Ferury [Cikering, 2002] Dvi Mxwell Cikering. Optiml struture ientifition wit greey ser. Tenil Report MSR-TR , Mirosoft Reser, Sumitte to JMLR. [Cowell et l., 1999] Roert Cowell, A. Pilip Dwi, Steffen L. Luritzen, n Dvi J. Spiegellter. Proilisti Networks n Expert Systems. Springer, New York, [Dor n Trsi, 1992] Dorit Dor n Miel Trsi. A simple lgoritm to onstrut onsistent extension of prtilly oriente grp. Tenil Report R-185, UCLA Cognitive Systems Lortory, [Kočk n Cstelo, 2001] Tomáš Kočk n Roert Cstelo. Improve lerning of Byesin networks. In Proeeings of Seventeent Conferene on Unertinty in Artifiil Intelligene. Morgn Kufmnn, [Perl, 1988] Jue Perl. Proilisti Resoning in Intelligent Systems. Morgn Kufmnn, Sn Mteo, Referenes [Anersson et l., 1999] Steen A. Anersson, Dvi Mign, n Miel D. Perlmn. A rteriztion of Mrkov equivlene lsses for yli igrps.