Bayesian belief networks: learning and inference

Transcription

1 CS 1675 Introducton to chn Lrnng Lctur 16 ysn lf ntworks: lrnng nd nfrnc los Huskrcht 5329 Snnott Squr Dt: Dnsty stton D { D1 D2.. Dn} D x vctor of ttrut vlus Ojctv: try to stt th undrlyng tru prolty dstruton ovr vrls p usng xpls n D tru dstruton n spls p D D D.. D } { 1 2 n stt pˆ Stndrd d ssuptons: Spls r ndpndnt of ch othr co fro th s dntcl dstruton fxd p 1

2 odlng coplx dstrutons Quston: How to odl nd lrn coplx ultvrt dstrutons pˆ wth lrg nur of vrls? xpl: odlng of dss syptos rltons Dss: pnuon tnt syptos fndngs l tsts: vr Cough lnss WC wht lood clls count Chst pn tc. odl of th full jont dstruton: nuon vr Cough lnss WC Chst pn On prolty pr ssgnnt of vlus to vrls: nuon = vr = Cought= WC=Hgh Chst pn= ysn lf ntworks Ns ysn lf ntworks lt 80s gnnng of 90s Gv solutons to th spc cquston ottlncks rtl solutons for t coplxts Ky fturs: Rprsnt th full jont dstruton ovr th vrls or copctly wth sllr nur of prtrs. k dvntg of condtonl nd rgnl ndpndncs ong rndo vrls nd Y r ndpndnt Y Y nd Y r condtonlly ndpndnt gvn Z Y Z Z Y Z Y Z Z 2

3 ysn lf ntwork 1. Drctd cyclc grph Nods = rndo vrls urglry rthquk lr ry clls nd ohn clls Lnks = drct cusl dpndncs twn vrls. h chnc of lr ng s nfluncd y rthquk h chnc of ohn cllng s ffctd y th lr urglry rthquk lr ohnclls ryclls ysn lf ntwork 2. Locl condtonl dstrutons rltng vrls nd thr prnts urglry rthquk lr ohnclls ryclls 3

4 ysn lf ntwork urglry ohnclls lr rthquk ryclls ull jont dstruton n Ns ull jont dstruton s dfnd n trs of locl condtonl dstrutons otnd v th chn rul: n xpl: 1.. n ssu th followng ssgnnt of vlus to rndo vrls hn ts prolty s: p 4

5 5 ull jont dstruton n Ns Rwrt th full jont prolty usng th product rul: # of prtrs of th full jont: rtr coplxty prol In th N th full jont dstruton s dfnd s: Wht dd w sv? lr xpl: nry ru ls vrls urglry ohnclls lr rthquk ryclls n n p On prtr s for fr: # of prtrs of th N:?

6 ysn lf ntwork: prtrs count otl: 20 urglry ohnclls lr rthquk ryclls rtr coplxty prol In th N th full jont dstruton s dfnd s: n p 1.. n Wht dd w sv? lr xpl: 5 nry ru ls vrls # of prtrs of th full jont: urglry On prtr s for fr: # of prtrs of th N: ohnclls On prtr n vry condtonl s for fr:? lr rthquk ryclls 6

7 ysn lf ntwork: fr prtrs urglry otl fr prs: 10 ohnclls lr rthquk ryclls = = rtr coplxty prol In th N th full jont dstruton s dfnd s: n p 1.. n Wht dd w sv? lr xpl: 5 nry ru ls vrls # of prtrs of th full jont: urglry On prtr s for fr: # of prtrs of th N: ohnclls On prtr n vry condtonl s for fr: lr rthquk ryclls 7

8 Ns xpls In vrous rs: Intllgnt usr ntrfcs crosoft roulshootng dgnoss of tchncl dvc dcl dgnoss: thfndr CSC unn QR-D Collortv fltrng ltry pplctons Insurnc crdt pplctons Dgnoss of cr ngn Dgnos th ngn strt prol 8

9 Cr nsurnc xpl rdct cl costs dcl llty sd on pplcton dt ICU lr ntwork 9

10 CCS Coputr-sd tnt Cs Sulton syst CCS- dvlopd y rkr nd llr t Unvrsty of ttsurgh 422 nods nd 867 rcs QR-D dcl dgnoss n ntrnl dcn prtt ntwork of dss/fndngs rltons 10

11 11 Nïv ys odl spcl spl ysn lf ntwork usd s gnrtv clssfr odl odl of pxy = px y py Clss vrl y py ttruts r ndpndnt gvn y Lrnng: rtrz odls of py nd ll px j y= L stts of th prtrs Clss y 1 x 2 x n x.. 1 y x p y p d j j x Nïv ys odl spcl spl ysn lf ntwork usd s gnrtv clssfr odl odl of pxy = px y py Clssfcton: gvn x slct th clss Slct th clss wth th xu postror Clculton of postror s n xpl of N nfrnc Rr: w cn clcult th prolts fro th full jont Clss Y 1 2 n.. k u d j j d j j k u u y x p u y p y x p y p u y p u y p y p y p y p x x x

12 Lrnng of N Lrnng. Lrnng of prtrs of condtonl prolts Lrnng of th ntwork structur Vrls: Osrvl vlus prsnt n vry dt spl Hddn thy vlus r nvr osrvd n dt ssng vlus vlus sots prsnt sots not Nxt: Lrnng of th prtrs of N Vlus for ll vrls r osrvl stton of prtrs of N Id: dcopos th stton prol for th full jont ovr lrg nur of vrls to st of sllr stton prols corrspondng to locl prnt-vrl condtonls. xpl: ssu r nry wth ru ls vlus Lrnng of = 4 stton prols == == == == ssupton tht nls th dcoposton: prtrs of condtonl dstrutons r ndpndnt 12

13 stts of prtrs of N wo ssuptons tht prt th dcoposton: Spl ndpndnc D Θ Θ u1 rtr ndpndnc n N p Θ D p D q 1 j1 D u rtrs of ch condtonl on for vry ssgnnt of vlus to prnt vrls cn lrnd ndpndntly j # of nods # of prnts vlus Lrnng of N prtrs. xpl. xpl: nuon nuon?? HWCnu n???? lnss vr Cough Hgh WC lnnu vrnu Coughnu??? 13

14 Lrnng of N prtrs. xpl. Dt D dffrnt ptnt css: l v Cou HW nu lnss vr nuon Cough Hgh WC stts of prtrs of N uch lk ultpl con toss or roll of dc prols. sllr lrnng prol corrsponds to th lrnng of xctly on condtonl dstruton xpl: vr nuon rol: How to pck th dt to lrn? 14

15 Lrnng of N prtrs. xpl. Lrn: vr nuon Stp 1: Slct dt ponts wth nuon= l v Cou HW nu lnss vr nuon Cough Hgh WC Lrnng of N prtrs. xpl. Lrn: Stp 1: vr nuon Ignor th rst l v Cou HW nu lnss vr nuon Cough Hgh WC 15

16 Lrnng of N prtrs. xpl. Lrn: vr nuon Stp 2: Slct vlus of th rndo vrl dfnng th dstruton of vr l v Cou HW nu lnss vr nuon Cough Hgh WC Lrnng of N prtrs. xpl. Lrn: vr nuon Stp 2: Ignor th rst v lnss vr nuon Cough Hgh WC 16

17 Lrnng of N prtrs. xpl. Lrn: vr nuon Stp 3: Lrnng th L stt v lnss vr nuon Cough Hgh WC vr nuon Lrnng of N prtrs. ysn lrnng. Lrn: vr nuon Stp 3: Lrnng th ysn postror ssu th pror nuon vr nuon ~ t34 v lnss vr Cough Hgh WC ostror: vr nuon ~ t66 stts 6 1 vr nuon

18 stts of prtrs of N uch lk ultpl con toss or roll of dc prols. sllr lrnng prol corrsponds to th lrnng of xctly on condtonl dstruton xpl: vr nuon rol: How to pck th dt to lrn? nswr: 1. Slct dt ponts wth nuon= gnor th rst 2. ocus on slct only vlus of th rndo vrl dfnng th dstruton vr 3. Lrn th prtrs of th locl condtonls th s wy s w lrnd th prtrs of sd con or d rolstc nfrncs N odls copctly th full jont dstruton y tkng dvntg of xstng ndpndncs twn vrls Splfs th rprsntton nd lrnng of odl Cn usd for th dffrnt nfrnc tsks. 18

19 ys thor Condtonl/jont prolty rltons. ys thor swtchs condtonng vnts : Whn s t usful? Whn w r ntrstd n coputng th dgnostc qury fro th cusl prolty ffct cus cus cus ffct ffct Rson: It s oftn sr to ssss cusl prolty.g. rolty of pnuon cusng fvr vs. prolty of pnuon gvn fvr xpl: spl dgnostc nfrnc Dvc qupnt oprtng norlly or lfunctonng. Oprton of th dvc snsd ndrctly v snsor Snsor rdng s thr Hgh or Low N Dvc sttus Dvc sttus norl lfunctonng Snsor rdng Snsor rdng Dvc sttus Sttus\Snsor Hgh Low norl lfunc

20 xpl: spl dgnostc nfrnc Dgnostc nfrnc: coput th prolty of dvc oprtng norlly or lfunctonng gvn snsor rdng Dvc sttus Snsor rdng hgh Dvc sttus norl Snsor rdng hgh Dvc sttus lfunct Snsor rdng hgh Not w hv th oppost condtonl prolts: thy r uch sr to stt Soluton: pply ys thor to rvrs th condtonng vrls Dvc sttus Snsor rdng xpl: spl dgnostc nfrnc Dvc qupnt oprtng norlly or lfunctonng. Oprton of th dvc snsd ndrctly v snsor Snsor rdng s thr Hgh or Low Dvc sttus Dvc sttus norl lfunctonng Snsor rdng Dvc sttus Snsor rdng Sttus\Snsor Hgh Low norl lfunc Dvc sttus Snsor rdng hgh? 20

21 ys thor ssu vrl wth ultpl vlus ys thor cn rwrttn s: 1 2 k j j j k j 1 j j Usd n prctc whn w wnt to coput: for ll vlus of 1 2 k j j. xpl: spl dgnostc nfrnc Dvc sttus Dvc sttus norl lfunctonng Snsor rdng Snsor rdng Dvc sttus Sttus\Snsor Hgh Low norl lfunc Dvc sttus Snsor rdng hgh? Dvc sttus nor Snsor rdng hgh Snsor rdng hghdvc sttus nor Snsor rdng hgh 21

22 xpl: spl dgnostc nfrnc. Dvc sttus Dvc sttus norl lfunctonng Snsor rdng Snsor rdng Dvc sttus Sttus\Snsor Hgh Low norl lfunc Dvc sttus Snsor rdng hgh? Dvc sttus nor Snsor rdng hgh Snsor rdng hghdvc sttus nor Snsor rdng hgh Snsor rdng hgh Dvc sttus nor Dvc sttus nor Snsor rdng hgh xpl: spl dgnostc nfrnc. Dvc sttus Dvc sttus norl lfunctonng Snsor rdng Snsor rdng Dvc sttus Sttus\Snsor Hgh Low norl lfunc Dvc sttus Snsor rdng hgh? Dvc sttus nor Snsor rdng hgh Snsor rdng hgh Dvc sttus nor Dvc sttus nor Snsor rdng hgh Snsor rdng hgh Dvc sttus nor Dvc sttus nor Snsor rdng hgh Dvc sttus lf Dvc sttus lf 22

23 Infrnc n ysn ntworks N odls copctly th full jont dstruton y tkng dvntg of xstng ndpndncs twn vrls Splfs th rprsntton nd lrnng of odl ut w r ntrstd n solvng vrous nfrnc tsks: Dgnostc tsk. fro ffct to cus urglry ohnclls rdcton tsk. fro cus to ffct ohnclls urglry Othr prolstc qurs qurs on jont dstrutons. lr n quston: Cn w tk dvntg of ndpndncs to construct spcl lgorths nd spdng up th nfrnc? Infrnc n ysn ntwork d nws: xct nfrnc prol n Ns s N-hrd Coopr pproxt nfrnc s N-hrd Dgu Luy ut vry oftn w cn chv sgnfcnt provnts ssu our lr ntwork urglry rthquk lr ohnclls ryclls ssu w wnt to coput: 23

24 24 Infrnc n ysn ntworks Coputng: pproch 1. lnd pproch. Su out ll un-nstnttd vrls fro th full jont xprss th jont dstruton s product of condtonls Coputtonl cost: Nur of ddtons:? Nur of products:? Infrnc n ysn ntworks Coputng: pproch 1. lnd pproch. Su out ll un-nstnttd vrls fro th full jont xprss th jont dstruton s product of condtonls Coputtonl cost: Nur of ddtons: 15 Nur of products:?

25 25 Infrnc n ysn ntworks Coputng: pproch 1. lnd pproch. Su out ll un-nstnttd vrls fro th full jont xprss th jont dstruton s product of condtonls Coputtonl cost: Nur of ddtons: 15 Nur of products: 16*4=64 Infrnc n ysn ntworks pproch 2. Intrlv sus nd products Cons sus nd product n srt wy ultplctons y constnts cn tkn out of th su Coputtonl cost: Nur of ddtons: 1+2*[1+1+2*1]=? Nur of products: 2*[2+2*1+2*1]=? ] [. ]] [ ][ [

26 26 Infrnc n ysn ntworks pproch 2. Intrlv sus nd products Cons sus nd product n srt wy ultplctons y constnts cn tkn out of th su Coputtonl cost: Nur of ddtons:? 1 Infrnc n ysn ntworks pproch 2. Intrlv sus nd products Cons sus nd product n srt wy ultplctons y constnts cn tkn out of th su Coputtonl cost: Nur of ddtons:? 2*1

27 27 Infrnc n ysn ntworks pproch 2. Intrlv sus nd products Cons sus nd product n srt wy ultplctons y constnts cn tkn out of th su Coputtonl cost: Nur of ddtons:? 2*2*1 Infrnc n ysn ntworks pproch 2. Intrlv sus nd products Cons sus nd product n srt wy ultplctons y constnts cn tkn out of th su Coputtonl cost: Nur of ddtons:? 2*2*1 2*1

28 28 Infrnc n ysn ntworks pproch 2. Intrlv sus nd products Cons sus nd product n srt wy ultplctons y constnts cn tkn out of th su Coputtonl cost: Nur of ddtons:? 2*2*1 2*1 2*1 1 Infrnc n ysn ntworks pproch 2. Intrlv sus nd products Cons sus nd product n srt wy ultplctons y constnts cn tkn out of th su Coputtonl cost: Nur of ddtons: 1+2*[1+1+2*1]=9 2*2*1 2*1 2*1 1

29 29 Infrnc n ysn ntworks pproch 2. Intrlv sus nd products Cons sus nd product n srt wy ultplctons y constnts cn tkn out of th su Coputtonl cost: Nur of products:? 1 Infrnc n ysn ntworks pproch 2. Intrlv sus nd products Cons sus nd product n srt wy ultplctons y constnts cn tkn out of th su Coputtonl cost: Nur of products:? 2*2 *2*1

30 30 Infrnc n ysn ntworks pproch 2. Intrlv sus nd products Cons sus nd product n srt wy ultplctons y constnts cn tkn out of th su Coputtonl cost: Nur of products:? 2*2 *2*1 2*2*1 2*2 Infrnc n ysn ntworks pproch 2. Intrlv sus nd products Cons sus nd product n srt wy ultplctons y constnts cn tkn out of th su Coputtonl cost: Nur of products: 2*[2+2*1+2*1]=16 2*2 *2*1 2*2*1 2*2

31 31 Infrnc n ysn ntworks pproch 2. Intrlv sus nd products Cons sus nd product n srt wy ultplctons y constnts cn tkn out of th su Coputtonl cost: Nur of ddtons: 1+2*[1+1+2*1]=9 Nur of products: 2*[2+2*1+2*1]=16. Vrl lnton Vrl lnton: Slr d ut ntrlv su nd products on vrl t th t durng nfrnc.g. Qury rqurs to lnt nd ths cn don n dffrnt ordr

32 32 Vrl lnton ssu ordr: to clcult Infrnc n ysn ntwork xct nfrnc lgorths: Vrl lnton Rcursv dcoposton Coopr Drwch Syolc nfrnc D roso lf propgton lgorth rl Clustrng nd jont tr pproch Lurtzn Spglhltr rc rvrsl Olstd Schchtr pproxt nfrnc lgorths: ont Crlo thods: orwrd splng Lklhood splng Vrtonl thods