C 188: Aificial Inelligence ping 2008 Bayes Nes 2/5/08, 2/7/08 Dan Klein UC Bekeley Bayes Nes A Bayes ne is an efficien encoding of a pobabilisic model of a domain Quesions we can ask: Infeence: given a fixed BN, wa is P( e)? epesenaion: given a fixed BN, wa kinds of disibuions can i encode? Modeling: wa BN is mos appopiae fo a given domain? Bayes Ne Bayes Ne emanics A Bayes ne: A se of nodes, one pe vaiable A dieced, acyclic gap A condiional disibuion of eac vaiable condiioned on is paens (e paameesθ) A 1 A n emanics: A BN defines a join pobabiliy disibuion ove is vaiables: Building e (Enie) Join : Alam Newok We can ake a Bayes ne and build any eny fom e full join disibuion i encodes ypically, ee s no eason o build ALL of i We build wa we need on e fly o empasize: evey BN ove a domain implicily epesens some join disibuion ove a domain, bu is specified by local pobabiliies 1
ize of a Bayes Ne How big is a join disibuion ove N Boolean vaiables? How big is an N-node ne if nodes ave k paens? Bo give you e powe o calculae BNs: Huge space savings! Also easie o elici local CPs Also uns ou o be fase o answe queies (nex class) Bayes Nes o fa: ow a Bayes ne encodes a join disibuion Nex: ow o answe queies abou a disibuion Key idea: condiional independence Las class: assembled BNs using an inuiive noion of condiional independence as causaliy oday: fomalize ese ideas Main goal: answe queies abou condiional independence and influence Afe a: ow o answe numeical queies (infeence) Condiional Independence eminde: independence and ae independen if : Independence Fo is gap, you can fiddle wi θ (e CPs) all you wan, bu you won be able o epesen any disibuion in wic e flips ae dependen! and ae condiionally independen given Z 1 2 (Condiional) independence is a popey of a disibuion All disibuions opology Limis Disibuions Independence in a BN Given some gap opology G, only ceain join disibuions can be encoded e gap sucue guaanees ceain (condiional) independences (ee mig be moe independence) Adding acs inceases e se of disibuions, bu as seveal coss Z Z Z Impoan quesion abou a BN: Ae wo nodes independen given ceain evidence? If yes, can calculae using algeba (eally edious) If no, can pove wi a coune example : Z Quesion: ae and Z independen? Answe: no necessaily, we ve seen examples oewise: low pessue causes ain wic causes affic. can influence Z, Z can influence (via ) Addendum: ey could be independen: ow? 2
Causal Cains Common Cause is configuaion is a causal cain Z : Low pessue : ain Z: affic Anoe basic configuaion: wo effecs of e same cause Ae and Z independen? Ae and Z independen given? Z Is independen of Z given?! Evidence along e cain blocks e influence Obseving e cause blocks influence beween effecs.! : Pojec due : Newsgoup busy Z: Lab full Common Effec e Geneal Case Las configuaion: wo causes of one effec (v-sucues) Ae and Z independen? : emembe e ballgame and e ain causing affic, no coelaion? ill need o pove ey mus be (omewok) Ae and Z independen given? No: emembe a seeing affic pu e ain and e ballgame in compeiion? is is backwads fom e oe cases Obseving e effec enables influence beween effecs. Z : aining Z: Ballgame : affic Any complex example can be analyzed using ese ee canonical cases Geneal quesion: in a given BN, ae wo vaiables independen (given evidence)? oluion: gap seac! eacabiliy eacabiliy (e Bayes Ball) ecipe: sade evidence nodes Aemp 1: if wo nodes ae conneced by an undieced pa no blocked by a saded node, ey ae condiionally independen Almos woks, bu no quie Wee does i beak? Answe: e v-sucue a doesn coun as a link in a pa unless saded D L B Coec algoim: ade in evidence a a souce node y o eac age by seac aes: pai of (node, pevious sae ) uccesso funcion: unobseved: o any cild o any paen if coming fom a cild obseved: Fom paen o paen If you can eac a node, i s condiionally independen of e sa node given evidence 3
L B D Causaliy? Vaiables: : aining : affic D: oof dips : I m sad Quesions: D Wen Bayes nes eflec e ue causal paens: Ofen simple (nodes ave fewe paens) Ofen easie o ink abou Ofen easie o elici fom expes BNs need no acually be causal omeimes no causal ne exiss ove e domain E.g. conside e vaiables affic and Dips End up wi aows a eflec coelaion, no causaion Wa do e aows eally mean? opology may appen o encode causal sucue opology only guaaneed o encode condiional independencies : affic Basic affic ne Le s muliply ou e join : evese affic evese causaliy? 1/4 3/4 3/4 1/4 3/16 1/16 6/16 6/16 9/16 7/16 1/3 2/3 3/16 1/16 6/16 6/16 1/2 1/2 1/7 6/7 4
: Coins Alenae BNs Exa acs don peven epesening independence, jus allow non-independence 1 2 1 2 ummay Infeence Bayes nes compacly encode join disibuions Guaaneed independencies of disibuions can be deduced fom BN gap sucue A Bayes ne may ave oe independencies a ae no deecable unil you inspec is specific disibuion e Bayes ball algoim (aka d-sepaaion) ells us wen an obsevaion of one vaiable can cange belief abou anoe vaiable Infeence: calculaing some saisic fom a join pobabiliy disibuion s: Poseio pobabiliy: Mos likely explanaion: D L B Infeence by Enumeaion eminde: Alam Newok P(sun)? P(sun wine)? P(sun wine, wam)? summe summe summe summe wine wine wine wine wam wam cold cold wam wam cold cold sun ain sun ain sun ain sun ain P 0.30 0.05 0.10 0.05 0.10 0.05 0.15 0.20 5
Infeence by Enumeaion Given unlimied ime, infeence in BNs is easy ecipe: ae e maginal pobabiliies you need Figue ou ALL e aomic pobabiliies you need Calculae and combine em : Wee did we use e BN sucue? We didn! Nomalizaion ick In is simple meod, we only need e BN o synesize e join enies Nomalize Infeence by Enumeaion Infeence by Enumeaion? Geneal case: Evidence vaiables: Quey vaiables: Hidden vaiables: All vaiables We wan: Fis, selec e enies consisen wi e evidence econd, sum ou H: Finally, nomalize e emaining enies o condiionalize Obvious poblems: Wos-case ime complexiy O(d n ) pace complexiy O(d n ) o soe e join disibuion 6
Nesing ums Vaiable Eliminaion: Idea Aomic infeence is exemely slow! ligly cleve way o save wok: Move e sums as fa ig as possible : Los of edundan wok in e compuaion ee We can save ime if we cace all paial esuls is is e basic idea beind vaiable eliminaion ampling Pio ampling Basic idea: Daw N samples fom a sampling disibuion Compue an appoximae poseio pobabiliy ow is conveges o e ue pobabiliy P Cloudy Ouline: ampling fom an empy newok ejecion sampling: ejec samples disageeing wi evidence Likeliood weiging: use evidence o weig samples pinkle WeGass ain Pio ampling is pocess geneaes samples wi pobabiliy i.e. e BN s join pobabiliy Le e numbe of samples of an even be en I.e., e sampling pocedue is consisen We ll ge a bunc of samples fom e BN: c, s,, w c, s,, w Cloudy C c, s,, w pinkle ain c, s,, w c, s,, w WeGass W If we wan o know P(W) We ave couns <w:4, w:1> Nomalize o ge P(W) = <w:0.8, w:0.2> is will ge close o e ue disibuion wi moe samples Can esimae anying else, oo Wa abou P(C )? P(C, w)? 7
ejecion ampling Likeliood Weiging Le s say we wan P(C) No poin keeping all samples aound Jus ally couns of C oucomes Le s say we wan P(C s) ame ing: ally C oucomes, bu ignoe (ejec) samples wic don ave =s is is ejecion sampling I is also consisen (coec in e limi) pinkle Cloudy C WeGass W c, s,, w c, s,, w c, s,, w c, s,, w c, s,, w ain Poblem wi ejecion sampling: If evidence is unlikely, you ejec a lo of samples ou don exploi you evidence as you sample Conside P(B a) Buglay Idea: fix evidence vaiables and sample e es Buglay Alam Alam Poblem: sample disibuion no consisen! oluion: weig by pobabiliy of evidence given paens Likeliood ampling Likeliood Weiging ampling disibuion if z sampled and e fixed evidence Cloudy Cloudy C pinkle ain Now, samples ave weigs W ain WeGass ogee, weiged sampling disibuion is consisen Likeliood Weiging Noe a likeliood weiging doesn solve all ou poblems ae evidence is aken ino accoun fo downseam vaiables, bu no upseam ones A bee soluion is Makov-cain Mone Calo (MCMC), moe advanced We ll eun o sampling fo obo localizaion and acking in dynamic BNs Cloudy C W ain 8