Question: Is there a BN that is a perfect map for a given MN?

Transcription

1 School of omuter Scence Probablstc rahcal Models ayesan & Markov Networks: unfed vew Recetor Knase ene Recetor X 1 X 2 Knase Knase E X 3 X 4 X 5 TF F X 6 ene 7 X 8 X H Erc Xng Lecture 3 January Readng: KF-cha4.5 Erc MU Queston: Is there a N that s a erfect ma for a gven MN? The "damond" MN Erc MU

2 Queston: Is there a N that s a erfect ma for a gven MN? {} {} {} {} {} {} {} Ths MN does not have a erfect I-ma as N! Erc MU Queston: Is there an MN that s a erfect I-ma to a gven N? V-structure eamle Erc MU

3 Queston: Is there an MN that s a erfect I-ma to a gven N? V-structure has no equvalent n MNs! Erc MU Mnmal I-mas Instead of attemtng erfect I-mas between Ns and MNs we can try mnmal I-mas Recall: H s a mnmal I-ma for f IH I Removal of a sngle edge n H renders t s not an I-ma Note: If H s a mnmal I-ma of H need not necessarly satsfy all the ndeendence relatonshs n Erc MU

4 Mnmal I-mas from Ns to MNs: Markov lanket Markov lanket of X n a N : M X s the unque mnmal set U of nodes n such that X all other nodes U s guaranteed to hold for any dstrbuton that factorzes over efn: M X s the set of nodes consstng of X s arents X s chldren and other arents of X s chldren Idea: The neghbors of X n H --- the mnmal I-ma of --- should be M X! Erc MU Mnmal I-mas from Ns to MNs: Moral rahs efn 5.7.3: The moral grah M of a N s an undrected grah that contans an undrected edge between X and Y f: there s a drected edge between them n ether drecton X and Y are arents of the same node omment: ths defnton ensures M X s the set of neghbors of X n M Erc MU

5 Mnmal I-mas from Ns to MNs: Moral grah s the mnmal I-ma orollary 5.7.4: The moral grah M of any N s a mnmal I- ma for Moralzaton turns each X PaX nto a fully connected subset Ps assocated wth the network can be used as clque otentals The moral grah loses some ndeendence nformaton ut all ndeendence roostons n the moral grah careful not ncludng non-ndeendence assumtons are wthheld n the N Erc MU Mnmal I-mas from Ns to MNs: Perfect I-mas Prooston 5.7.5: If the N s "moral" then ts moralzed grah M s a erfect I-ma of. Proof sketch: IM I from before The only ndeendence relatons that are otentally lost from to M are those arsng from V-structures Snce has no V-structures t s moral no ndeendences are lost n M Eamle of M s a erfect I-ma of? Erc MU

6 Soundness of d-searaton Recall d-searaton Let U ={X Y Z} be three dsjont sets of nodes n a N. Let + be the ancestral grah: the nduced N over U ancestorsu. Then d-se X;YZ ff se M + X;YZ -se ;I L -se ;I S se M + ;I L Se M + ;I S Erc MU Soundness of d-searaton Why t works: : M: M + : Idea: Informaton blocked through common chldren n that are not n the condtonng varables s smulated n M+ by gnorng all chldren. Erc MU

7 Mnmal I-mas from Ns to MNs: Summary Moral rah M s a mnmal I-ma of If s moral then M s a erfect I-ma of -se X;YZ se M + X;YZ Net: mnmal I-mas from MNs to Ns Erc MU Mnmal I-mas from MNs to Ns: ny N I-ma for an MN must add trangulatng edges nto the grah Erc MU

8 Mnmal I-mas from MNs to Ns: chordal grahs efn : Let X 1 -X 2 - X k -X 1 be a loo n a grah. chord n a loo s an edge connectng X and X j fo non-consecutve {X X j } n undrected grah H s chordal f any loo X 1 -X 2 - X k -X 1 for K >= 4 has a chord efn : drected grah s chordal f ts underlyng undrected grah s chordal Erc MU Mnmal I-mas from MNs to Ns: trangulaton Thm : Let H be an MN and be any N mnmal I- ma for H. Then can have no mmoraltes. Intutve reason: Immoraltes ntroduce addtonal ndeendences that are not n the orgnal MN cf. roof for theorem n K&F orollary : Let K be any mnmal I-ma for H. Then K s necessarly chordal! ecause any non-trangulated loo of length at least 4 n a ayesan network grah necessarly contans an mmoralty Process of addng edges also called trangulaton Erc MU

9 Thm : Let H be a non-chordal MN. Then there s no N that s a erfect I-ma for H. Proof: Mnmal N I-ma for MN H s chordal It must therefore have addtonal drected edges not resent n H Each addtonal edge elmnates some ndeendence assumtons Hence roved. Erc MU lque trees 1 Notaton: Let H be a connected undrected grah. Let 1 k be the set of mamal clques n H. Let T be a tree structured grah whose nodes are 1 k. Let j be two clques n the tree connected by an edge. efne S j = j be the se-set between and j Let W <j = Varables VarablesS j --- the resdue set Erc MU

10 lque trees 2 tree T s a clque tree for H f: Each node n T corresonds to a clque n H and each mamal clque n H s a node n T Each seset S j searates W <j and W <j Every undrected chordal grah H has a clque tree T. Proof by nducton cf. Theorem n K&F Eamle n net slde Erc MU Eamle Eamle chordal grah and ts clque tree E E E F EF E F E Erc MU

11 I-mas of MN as N: Thm : Let H be a chordal MN. Then there ests a N such that IH = I. Proof sketch: Snce H s an MN t has a clque tree Number the nodes consstent wth clque orderng E F E E EF Erc MU I-mas of MN as N: Thm : Let H be a chordal MN. Then there ests a N such that IH = I. Proof sketch contd: For each node X let k be the frst clque t occurs n. efne PaX = var{ k } X {X 1 X -1 } F E E and H have the same edges ll arents of each X are n the same clque node they are connected no mmoraltes n EF E Erc MU

12 Mnmal I-mas from MNs to Ns: Summary mnmal I-ma N of an MN s chordal Obtaned by trangulatng g the MN If the MN s chordal there s a erfect N I-ma for the MN Obtaned from the corresondng clque-tree Erc MU Summary Investgated the relatonsh between Ns and MNs They reresent dfferent famles of ndeendence assumtons hordal grahs can be reresented n both Not mentoned: han networks suerset of both Ns and MNs Why we care about ths: N and MN offer dfferent semantcs for desgner to cature or eresson condtonal ndeendences among varables Under certan condton N can be reresented as an MN and vce versa In the future for certan oeraton.e. nference we wll be usng a sngle reresentaton as the data structure for whch an algorthm can oerate on. Ths makes algorthm desgn and analyss of the algorthms smler Erc MU

13 Where s the grah structure come from? The goal: ven set of ndeendent d samles assgnments t of random varables fnd the best the most lkely? grahcal model toology E R E ER=TFFTF ER=TFTTF.. ER=FTTTF R Erc MU R E ata ML Structural Learnng for comletely observed Ms 1 K 1 1 n 2 K 2 1 n K M K M 1 n Erc MU

14 14 Informaton Theoretc Interretaton of ML = log ; θ θ l = = = n n n n n n M count M log log log g ; θ θ θ = M log ˆ θ From sum over data onts to sum over count of varable states Erc MU Informaton Theoretc Interretaton of ML con'd = ˆ log ; θ θ l = = = M M M M ˆ log ˆ ˆ ˆ ˆ log ˆ ˆ ˆ ˆ ˆ log ˆ ˆ log ˆ g ; θ θ θ = H M I M ˆ ˆ ecomosable score and a functon of the grah structure Erc MU

15 Structural Search How many grahs over n nodes? O 2 n2 How many trees over n nodes? On! ut t turns out that we can fnd eact soluton of an otmal tree under MLE! Trck: n a tree each node has only one arent! how-lu algorthm Erc MU how-lu tree learnng algorthm Objecton functon: l θ ; = log ˆ θ = M how-lu: Iˆ M Hˆ = M Iˆ For each ar of varable and j omute emrcal dstrbuton: omute mutual nformaton: count j ˆ X X j = M Iˆ X X = j ˆ ˆ j j log j ˆ ˆ j efne a grah wth node 1 n Edge Ij gets weght Iˆ X X j Erc MU

16 how-lu algorthm con'd Objecton functon: l θ ; = log ˆ θ = M Iˆ M Hˆ = M Iˆ how-lu: Otmal tree N omute mamum weght sannng tree recton n N: ck any node as root do breadth-frst-search to defne drectons I-equvalence: E E E = I + I + I + I E Erc MU Structure Learnng for general grahs Theorem: The roblem of learnng a N structure wth at most d arents s NP-hard for any fed d 2 Most structure learnng aroaches use heurstcs Elot score decomoston Two heurstcs that elot decomoston n dfferent ways reedy search through sace of node-orders Local search of grah structures Erc MU

17 Learnng rahcal Model Structure va Neghborhood Selecton Erc MU Inferrng gene regulatory networks Network of cs-regulatory athways Success stores n sea urchn frut fly etc from decades of eermental research Statstcal modelng and automated learnng just started Erc MU

18 Undrected rahcal Models Why? Sometmes an UNIRETE assocaton grah makes more sense and/or s more nformatve gene eressons may be nfluenced by unobserved factor that are osttranscrtonally regulated The unavalablty of the state of results n a constran over and Erc MU aussan rahcal Models Multvarate aussan densty: 1 1 T 1 µ Σ = e{ - - µ Σ - µ } n / 2 1/ Σ WOL: let 1/ L - / Q 1 µ = 0 Q = e 2 q n / 2 2 < j 2 qj j We can vew ths as a contnuous Markov Random Feld wth otentals defned on every node and edge: Erc MU

19 Parwse MRF e.g. Isng Model ssumng the nodes are dscrete and edges are weghted then for a samle d d we have Erc MU The covarance and the recson matrces ovarance matr rahcal model nterretaton? Precson matr rahcal model nterretaton? Erc MU

20 Sarse recson vs. sarse covarance n M Σ = Σ = X 1 Σ 15 = 0 X1 X 5 X nbrs 1 or nbrs5 1 X 5 Σ15 = 0 Erc MU nother eamle How to estmate ths MRF? What f >> n MLE does not est n general! What about only learnng a sarse grahcal model? Ths s ossble when s=on Very often t s the structure of the M that s more nterestng Erc MU

21 Recall lasso Erc MU rah Regresson Neghborhood selecton Lasso: Erc MU

22 rah Regresson Erc MU rah Regresson It can be shown that: gven d samles and under several techncal condtons e.g. "rreresentable" the recovered structured s "sarsstent" even when >> n Erc MU

23 onsstency Theorem: for the grahcal regresson algorthm under certan verfable condtons omtted here for smlcty: Note the from ths theorem one should see that the regularzer s not actually used to ntroduce an artfcal sarsty bas but a devse to ensure consstency under fnte data and hgh dmenson condton. Erc MU Learnng Isng Model.e. arwse MRF ssumng the nodes are dscrete and edges are weghted then for a samle d d we have It can be shown followng the same logc that we can use L_1 regularzed logstc regresson to obtan a sarse estmate of the neghborhood of each varable n the dscrete case. Erc MU

24 Summary The how-lu lgorthm ML tree structure Other structures? rahcal aussan Model The recson matr encode structure Not estmatable when >> n Neghborhood selecton: ondtonal dst under M Pseudo-lkelhood under Isng model rahcal lasso Sarsstency Erc MU