Bayesian belief networks

Transcription

1 Lecture 14 ayesa belef etworks los Hauskrecht 5329 Seott Square Desty estmato Data: D { D1 D2.. D} D x a vector of attrbute values ttrbutes: modeled by radom varables { 1 2 d} wth: otuous values Dscrete values.g. temperature wth umercal values or chest pa wth dscrete values [o-pa mld moderate strog] Uderlyg true probablty dstrbuto: p 1

2 Data: Desty estmato D { D1 D2.. D} D x a vector of attrbute values Objectve: try to estmate the uderlyg true probablty dstrbuto over varables p usg examples D true dstrbuto samples p D D D.. D } { 1 2 estmate pˆ Stadard d assumptos: Samples are depedet of each other come from the same detcal dstrbuto fxed p Learg va parameter estmato I ths lecture we cosder parametrc desty estmato asc settgs: set of radom varables { 1 2 d} model of the dstrbuto over varables wth parameters : pˆ Data D D D.. D } { 1 2 Objectve: fd the parameters the best that expla the observed data 2

3 arameter estmato axmum lkelhood L maxmze p D yelds: oe set of parameters L the target dstrbuto s approxmated as: pˆ p Θ L ayesa parameter estmato uses the posteror dstrbuto over possble parameters p D p p D p D Yelds: all possble settgs of ad ther weghts he target dstrbuto s approxmated as: p ˆ p D p Θ p Θ D dθ Θ arameter estmato Other possble crtera: axmum a posteror probablty maxmze p Θ D mode of the posteror Yelds: oe set of parameters Θ pproxmato: pˆ p Θ xpected value of the parameter Θˆ Θ mea of the posteror xpectato take wth regard to posteror p Θ D Yelds: oe set of parameters pproxmato: pˆ p Θˆ 3

4 Desty estmato So far we have covered desty estmato for smple dstrbuto models: eroull omal ultomal Gaussa osso ut what f: he dmeso of { 1 2 d} s large xample: patet data ompact parametrc dstrbutos do ot seem to ft the data.g.: multvarate Gaussa may ot ft We have oly a small umber of examples to do accurate parameter estmates How to lear complex dstrbutos How to lear complex multvarate dstrbutos pˆ umber of varables? wth large Oe soluto: Decompose the dstrbuto usg codtoal depedece relatos Decompose the parameter estmato problem to a set of smaller parameter estmato tasks Decomposto of dstrbutos uder codtoal depedece assumpto s the ma dea behd ayesa belef etworks 4

5 xample roblem descrpto: Dsease: peumoa atet symptoms fdgs lab tests: Fever ough aleess W whte blood cells cout hest pa etc. Represetato of a patet case: Symptoms ad dsease are represeted as radom varables Our objectves: Descrbe a multvarate dstrbuto represetg the relatos betwee symptoms ad dsease Desg of ferece ad learg procedures for the multvarate model odelg ucertaty wth probabltes Full jot dstrbuto: ssume { 1 2 d} are all radom varables that defe the doma Full jot: or 1 2 d Full jot t s suffcet to do ay type of probablstc ferece: omputato of jot probabltes for sets of varables omputato of codtoal probabltes 1 2 rue 3 False 5

6 argalzato Jot probablty dstrbuto for a set varables Defes probabltes for all possble assgmets to values of varables the set peumoa Wcout eumoa rue False 23 table Wcout hgh ormal low eumoa Wcout argalzato summg of rows or colums - summg out varables Varable depedece he jot dstrbuto over a subset of varables ca be always computed from the jot dstrbuto through margalzato How about the opposte? a we recover the jot from the jot over subsets? peumoa Wcout Wcout eumoa hgh ormal low eumoa rue False Wcout??????

7 Varable depedece he jot dstrbuto over a subset of varables ca be always computed from the jot dstrbuto through margalzato a we recover the jot from the jot over subsets? NO! Oly excepto: whe varables are depedet peumoa Wcout Wcout eumoa hgh ormal low eumoa rue False Wcout?????? odtoal probablty : robablty of gve odtoal probablty odtoal probablty s defed terms of jot probabltes Jot probabltes ca be expressed terms of codtoal probabltes product rule odtoal probablty s useful for varous probablstc fereces eumoa rue Fever rue Wcout hgh ough rue cha rule 7

8 8 Iferece y query ca be computed from the full jot dstrbuto!!! Jot over a subset of varables s obtaed through margalzato odtoal probablty over a set of varables gve other varables values s obtaed through margalzato ad defto of codtoals j d j D c b a c a j j d D c b a d D c b a c a d D c a c a d D Iferece y jot probablty ca be expressed as a product of codtoals va the cha rule. It s ofte easer to defe the dstrbuto terms of codtoal probabltes:.g eumoa Fever F eumoa Fever

9 odelg ucertaty wth probabltes Full jot dstrbuto: jot dstrbuto over all radom varables defg the doma t s suffcet to represet the complete doma ad to do ay type of probablstc fereces roblems: Space complexty. o store full jot dstrbuto requres to remember Od umbers. umber of radom varables d umber of values Iferece complexty. o compute some queres requres. Od steps. cqusto problem. Who s gog to defe all of the probablty etres? eumoa example. omplextes. Space complexty. eumoa 2 values: F Fever 2: F ough 2: F Wcout 3: hgh ormal low paleess 2: F Number of assgmets: 2*2*2*3*2=48 We eed to defe at least 47 probabltes. me complexty. ssume we eed to compute the probablty of eumoa= from the full jot eumoa Fever ough F j F kh l u F Sum over 2*2*3*2=24 combatos j Wcout k ale u 9

10 ayesa belef etworks Ns ayesa belef etworks. Represet the full jot dstrbuto over the varables more compactly wth a smaller umber of parameters. ake advatage of codtoal ad margal depedeces amog radom varables ad are depedet ad are codtoally depedet gve larm system example ssume your house has a alarm system agast burglary. You lve the sesmcally actve area ad the alarm system ca get occasoally set off by a earthquake. You have two eghbors ary ad Joh who do ot kow each other. If they hear the alarm they call you but ths s ot guarateed. We wat to represet the probablty dstrbuto of evets: urglary arthquake larm ary calls ad Joh calls ausal relatos urglary arthquake larm Johalls aryalls 10

11 ayesa belef etwork 1. Drected acyclc graph Nodes = radom varables urglary arthquake larm ary calls ad Joh calls Lks = drect causal depedeces betwee varables. he chace of larm beg s flueced by arthquake he chace of Joh callg s affected by the larm urglary arthquake larm Johalls J aryalls ayesa belef etwork 2. Local codtoal dstrbutos relatg varables ad ther parets urglary arthquake larm J Johalls aryalls 11

12 ayesa belef etwork urglary Johalls F F larm J F F arthquake F F F F F aryalls F F Full jot dstrbuto Ns Full jot dstrbuto s defed terms of local codtoal dstrbutos obtaed va the cha rule: xample: 1.. ssume the followg assgmet of values to radom varables J F he ts probablty s: J F pa J F J 12

13 ayesa belef etworks Ns ayesa belef etworks Represet the full jot dstrbuto over the varables more compactly usg the product of local codtoals. ut how dd we get to local parameterzatos? swer: Graphcal structure ecodes codtoal ad margal depedeces amog radom varables ad are depedet ad are codtoally depedet gve he graph structure mples the decomposto!!! Idepedeces Ns 3 basc depedece structures: urglary urglary arthquake larm larm larm Johalls aryalls Johalls 13

14 Idepedeces Ns urglary urglary arthquake larm larm larm Johalls aryalls Johalls 1. Johalls s depedet of urglary gve larm J J J J Idepedeces Ns urglary urglary arthquake larm larm larm Johalls aryalls Johalls 2. urglary s depedet of arthquake ot kowg larm urglary ad arthquake become depedet gve larm!! 14

15 Idepedeces Ns urglary urglary arthquake 3. larm larm larm Johalls aryalls Johalls 3. aryalls s depedet of Johalls gve larm J J J J Idepedece N N dstrbuto models may codtoal depedece relatos relatg dstat varables ad sets hese are defed terms of the graphcal crtero called d- separato D-separato the graph Let Y ad Z be three sets of odes If ad Y are d-separated by Z the ad Y are codtoally depedet gve Z D-separato : s d-separated from gve f every udrected path betwee them s blocked wth ath blockg 3 cases that expad o three basc depedece structures 15

16 Udrected path blockg s d-separated from gve f every udrected path betwee them s blocked Udrected path blockg s d-separated from gve f every udrected path betwee them s blocked 16

17 Udrected path blockg s d-separated from gve f every udrected path betwee them s blocked 1. ath blockg wth a lear substructure Z Z Y Y Udrected path blockg s d-separated from gve f every udrected path betwee them s blocked 2. ath blockg wth the wedge substructure Z Z Y Y 17

18 Udrected path blockg s d-separated from gve f every udrected path betwee them s blocked 3. ath blockg wth the vee substructure Z Y Y Z or ay of ts descedats ot Idepedeces Ns urglary arthquake larm RadoReport Johalls aryalls arthquake ad urglary are depedet gve aryalls F urglary ad aryalls are depedet ot kowg larm F urglary ad RadoReport are depedet gve arthquake urglary ad RadoReport are depedet gve aryalls F 18

19 19 Full jot dstrbuto Ns J F J F F J F J F F Rewrte the full jot probablty usg the product rule: Full jot dstrbuto Ns J F J F J F F J F J F F Rewrte the full jot probablty usg the product rule:

20 arameters: full jot:? arameter complexty problem I the N the full jot dstrbuto s expressed as a product of codtoals of smaller complexty pa urglary arthquake N:? larm Johalls aryalls arameters: full jot: arameter complexty problem I the N the full jot dstrbuto s expressed as a product of codtoals of smaller complexty pa urglary arthquake N: larm arameters to be defed: full jot: Johalls aryalls N: