Probabilistic Graphical Models

Transcription

1 Shool o Comuer Sene Probbls Grhl Models Mmum lkelhood lernng o undreed GM Er Xng Leure 8 Februry 0 04 Redng: MJ Ch 9 nd Er CMU

2 Undreed Grhl Models Why? Somemes n UNDIRECTED ssoon grh mkes more sense nd/or s more normve gene eressons my be nluened by unobserved or h re osrnsronlly reguled A B C A B The unvlbly o he se o B resuls n onsrn over A nd C C A B C Er CMU

3 ML Sruurl Lernng v Neghborhood Seleon or omleely observed D MRF n n M M n Er CMU

4 Gussn Grhl Models Mulvre Gussn densy: WOLG: le We n vew hs s onnuous Mrkov Rndom Feld wh oenls dened on every node nd edge: e / / T n j j j n q q Q Q / / - e 0 4 Er CMU

5 Prwse MRF e.g. Isng Model Assumng he nodes re dsree nd edges re weghed hen or smle d we hve Er CMU

6 The ovrne nd he reson mres Covrne mr Grhl model nerreon? Preson mr Grhl model nerreon? Er CMU

7 Srse reson vs. srse ovrne n GGM X 5 0 X X 5 X nbrs or nbrs5 X Er CMU

8 Anoher emle How o esme hs MRF? Wh >> n MLE does no es n generl! Wh bou only lernng srse grhl model? Ths s ossble when s=on Very oen s he sruure o he GM h s more neresng Er CMU

9 Rell lsso Er CMU

10 Grh Regresson Neghborhood seleon Lsso: Er CMU

11 Grh Regresson Er CMU

12 Grh Regresson I n be shown h: gven d smles nd under severl ehnl ondons e.g. "rreresenble" he reovered sruured s "srssen" even when >> n Er CMU

13 Lernng Isng Model.e. rwse MRF Assumng he nodes re dsree nd edges re weghed hen or smle d we hve I n be shown ollowng he sme log h we n use L_ regulrzed logs regresson o obn srse esme o he neghborhood o eh vrble n he dsree se. Er CMU

14 Consseny Theorem: or he grhl regresson lgorhm under ern verble ondons omed here or smly: Noe he rom hs heorem one should see h he regulrzer s no ully used o nrodue n rl srsy bs bu devse o ensure onsseny under ne d nd hgh dmenson ondon. Er CMU

15 X X X X 4 ML Prmeer Es. or omleely observed MRFs o gven sruure The d: { z z z... z N N } Er CMU

16 Re: MLE or BNs Assumng he rmeers or eh CPD re globlly ndeenden nd ll nodes re ully observed hen he log-lkelhood unon deomoses no sum o lol erms one er node: l ; D log D log n log n n n ML jk n j ' k jk n j' k Er CMU

17 MLE or undreed grhl models For dreed grhl models he log-lkelhood deomoses no sum o erms one er mly node lus rens. For undreed grhl models he log-lkelhood does no deomose beuse he normlzon onsn s unon o ll he rmeers P n C In generl we wll need o do nerene.e. mrgnlzon o lern rmeers or undreed models even n he ully observed se. n C Er CMU

18 Log Lkelhood or UGMs wh bulr lque oenls Suen sss: or UGM VE he number o mes h onguron.e. X V = s observed n dse D={ N } n be reresened s ollows: In erms o he ouns he log lkelhood s gven by: There s nsy log n he lkelhood V m m m n n \ lque oun nd oun ol de de N m m D D n n n n n n log log log log log log l 8 Er CMU

19 Log Lkelhood or UGMs wh bulr lque oenls Suen sss: or UGM VE he number o mes h onguron.e. X V = s observed n dse D={ N } n be reresened s ollows: m de n ol oun m In erms o he ouns he log lkelhood s gven by: nd n V \ de m lque oun log D m log N log There s nsy log n he lkelhood Er CMU

20 Dervve o log Lkelhood Log-lkelhood: Frs erm: Seond erm: N m log log l m l log d d d d d d d d d Se he vlue o vrbles o 0 Er CMU

21 Condons on Clque Mrgnls Dervve o log-lkelhood l m N Hene or he mmum lkelhood rmeers we know h: * MLE m N In oher words he mmum lkelhood seng o he rmeers or eh lque he model mrgnls mus be equl o he observed mrgnls emrl ouns. Ths doesn ell us how o ge he ML rmeers jus gves us ondon h mus be ssed when we hve hem. de Er CMU

22 MLE or undreed grhl models Is he grh deomosble rnguled? Are ll he lque oenls dened on mml lques no sub-lques? e.g. 4 no X X X X X X 4 X X 4 Are he lque oenls ull bles or Gussns or rmeerzed more omly e.g. e? k k Er CMU

23 Proeres on MLE o lque oenls For deomosble models where oenls re dened on mml lques he MLE o lque oenls eque o he emrl mrgnls or ondonls o he orresondng lque. Thus he MLE n be solved by nseon!! I he grh s non-deomosble nd or he oenls re dened on non-mml lques e.g. 4 we ould no eque MLE o lques oenls o emrl mrgnls or ondonls. Poenl eressed s bulr orm: IPF Feure-bsed oenls: GIS Er CMU

24 MLE or deomosble undreed models Deomosble models: G s deomosble G s rnguled G hs junon ree Poenl bsed reresenon: Consder hn X X X. The lques re X X nd X X ; he seror s X The emrl mrgnls mus equl he model mrgnls. Le us guess h We n very h suh guess sses he ondons: nd smlrly MLE MLE MLE MLE s s s 4 Er CMU

25 MLE or deomosble undreed models on. Le us guess h To omue he lque oenls jus eque hem o he emrl mrgnls or ondonls.e. he seror mus be dvded no one o s neghbors. Then =. One more emle: X X 4 X X 4 4 MLE MLE MLE MLE MLE MLE 5 Er CMU

26 Non-deomosble nd/or wh non-mml lque oenls I he grh s non-deomosble nd or he oenls re dened on non-mml lques e.g. 4 we ould no eque emrl mrgnls or ondonls o MLE o lques oenls. X X 4 X X X X 4 X X } { j j j 4 / / s.. MLE j j j j j j j Homework! 6 Er CMU

27 MLE or undreed grhl models Is he grh deomosble rnguled? Are ll he lque oenls dened on mml lques no sub-lques? e.g. 4 no X X X X X X 4 X X 4 Are he lque oenls ull bles or Gussns or rmeerzed more omly e.g. e? k k Deomosble? M lque? Tbulr? Mehod Dre - - IPF Grden GIS Er CMU

28 Ierve Prooronl Fng IPF From he dervve o he lkelhood: we n derve noher relonsh: n whh ers mlly n he model mrgnl. Ths s hereore ed-on equon or. Solvng n losed-orm s hrd beuse ers on boh sdes o hs ml nonlner equon. The de o IPF s o hold ed on he rgh hnd sde boh n he numeror nd denomnor nd solve or on he le hnd sde. We yle hrough ll lques hen ere: N m l Need o do nerene here 8 Er CMU

29 Proeres o IPF Udes IPF eres se o ed-on equons: However we n rove s lso oordne sen lgorhm oordnes = rmeers o lque oenls. Hene eh se wll nrese he log-lkelhood nd wll onverge o globl mmum. I-rojeon: ndng dsrbuon wh he orre mrgnls h hs he mml enroy Er CMU

30 KL Dvergene Vew IPF n be seen s oordne sen n he lkelhood usng he wy o eressng lkelhoods usng KL dvergenes. We n show h mmzng he log lkelhood s equvlen o mnmzng he KL dvergene ross enroy rom he observed dsrbuon o he model dsrbuon: Usng roery o KL dvergene bsed on he ondonl hn rule: = b : log mn m KL l b b b b b b b b b b b b b b q q q q q q q q q q q q q q log log log KL KL KL 0 Er CMU

31 IPF mnmzes KL dvergene Pung hngs ogeher we hve KL KL KL I n be shown h hngng he lque oenl hs no ee on he ondonl dsrbuon so he seond erm n uneed. To mnmze he rs erm we se he mrgnl o he observed mrgnl jus s n IPF. Noe h hs s only good when he model s deomosble! We n nerre IPF udes s renng he old ondonl robbles - whle relng he old mrgnl robbly wh he observed mrgnl. Er CMU

32 MLE or undreed grhl models Is he grh deomosble rnguled? Are ll he lque oenls dened on mml lques no sub-lques? e.g. 4 no X X X X X X 4 X X 4 Are he lque oenls ull bles or Gussns or rmeerzed more omly e.g. e? k k Deomosble? M lque? Tbulr? Mehod Dre - - IPF Grden GIS Er CMU

33 Feure-bsed Clque Poenls So r we hve dsussed he mos generl orm o n undreed grhl model n whh lques re rmeerzed by generl bulr oenl unons. Bu or lrge lques hese generl oenls re eonenlly osly or nerene nd hve eonenl numbers o rmeers h we mus lern rom lmed d. One soluon: hnge he grhl model o mke lques smller. Bu hs hnges he deendenes nd my ore us o mke more ndeendene ssumons hn we would lke. Anoher soluon: kee he sme grhl model bu use less generl rmeerzon o he lque oenls. Ths s he de behnd eure-bsed models. Er CMU

34 Feures Consder lque o rndom vrbles n UGM e.g. hree onseuve hrers n srng o Englsh e. How would we buld model o? I we use sngle lque unon over he ull jon lque oenl would be huge: 6 rmeers. However we oen know h some rulr jon sengs o he vrbles n lque re que lkely or que unlkely. e.g. ng e on?ed qu? jk zzz... A eure s unon whh s vuous over ll jon sengs ee ew rulr ones on whh s hgh or low. For emle we mgh hve ng whh s he srng s ng nd 0 oherwse nd smlr eures or?ed e. We n lso dene eures when he nus re onnuous. Then he de o ell on whh s ve dsers bu we mgh sll hve om rmeerzon o he eure. Er CMU

35 Feures s Mrooenls By eonenng hem eh eure unon n be mde no mrooenl. We n mully hese mrooenls ogeher o ge lque oenl. Emle: lque oenl ould be eressed s: ngng?ed?ed e e e K k k k Ths s sll oenl over 6 ossble sengs bu only uses K rmeers here re K eures. By hvng one ndor unon er ombnon o we reover he sndrd bulr oenl. Er CMU

36 Combnng Feures Eh eure hs wegh k whh reresens he numerl srengh o he eure nd wheher nreses or dereses he robbly o he lque. The mrgnl over he lque s generlzed eonenl mly dsrbuon ully GLIM: ng e qu? ng qu? In generl he eures my be overlng unonsrned ndors or ny unon o ny subse o he lque vrbles: de ek k I How n we ombne eure no robbly model??ed zzz?ed zzz Er CMU

37 Feure Bsed Model We n mully hese lque oenls s usul: However n generl we n orge bou ssong eures wh lques nd jus use smled orm: Ths s jus our rend he eonenl mly model wh he eures s suen sss! Lernng: rell h n IPF we hve No obvous how o use hs rule o ude he weghs nd eures ndvdully!!! I k k e e 7 Er CMU

38 MLE o Feure Bsed UGMs Sled lkelhood unon Insed o omzng hs objeve drely we k s lower bound The logrhm hs lner uer bound Ths bound holds or ll n rulr or Thus we hve n n N N D D log log / ; ; l l log log log D log ; l 8 Er CMU

39 Generlzed Ierve Slng GIS Lower bound o sled loglkelhood Dene Rel gn Assume Convey o eonenl: We hve: D log ; l de log e log e e log e ; D l e e log e ; de D l 0 9 Er CMU

40 GIS Lower bound o sled loglkelhood Tke dervve: Se o zero where s he unnormlzed verson o Ude log e ; de D l e e e Rell IPF: 40 Er CMU

41 Summry IPF s generl lgorhm or ndng MLE o UGMs. ed-on equon or over sngle lques oordne sen I-rojeon n he lque mrgnl se Requres he oenl o be ully rmeerzed The lque desrbed by he oenls do no hve o be m-lque For ully deomosble model redues o sngle se eron GIS Ierve slng on generl UGM wh eure-bsed oenls IPF s sel se o GIS whh he lque oenl s bul on eures dened s n ndor unon o lque ongurons. IPF: GIS: log 4 Er CMU

42 Where does he eonenl orm ome rom? Revew: Mmum Lkelhood or eonenl mly.e. A ML esme he eeons o he suen sss under he model mus mh emrl eure verge. N m m m D log log log ; l N m N m D log ; l N m 4 Er CMU

43 Mmum Enroy We n roh he modelng roblem rom n enrely deren on o vew. Begn wh some ed eure eeons: Assumng eeons re onssen here my es mny dsrbuons whh ssy hem. Whh one should we sele? The mos unern or leble one.e. he one wh mmum enroy. Ths yelds new omzon roblem: m s.. H log Ths s vronl denon o dsrbuon! Er CMU

44 Soluon o he MEn Problem To solve he MEn roblem we use Lgrnge mullers: So eure onsrns + MEn eonenl mly. Problem s srly onve w.r.. so soluon s unque. L log e e L e sne e e log * * 44 Er CMU

45 A more generl MEn roblem h h h s.. log H log KL mn de h e 45 Er CMU

46 Consrns rom D Where do he onsrns ome rom? Jus s beore mesure he emrl ouns on he rnng d: m Ths lso ensures onsseny uomlly. Known s he mehod o momens... lw o lrge numbers We hve seen se o onve duly: In one se we ssume eonenl mly nd show h ML mles model eeons mus mh emrl eeons. In he oher se we ssume model eeons mus mh emrl eure ouns nd show h MEn mles eonenl mly dsrbuon. No duly g yeld he sme vlue o he objeve N Er CMU

47 Geomer nerreon All eonenl mly dsrbuon: E : h e All dsrbuons ssyng momen onsrns M : Pyhgoren heorem KL q KLq KL M M MEn : mn s.. KL KL q q h M q h KLq KL h M M MLk : mn s.. KL Er CMU KL q E KL KL M M 47

48 Summry Eonenl mly dsrbuon n be vewed s he soluon o n vronl eresson --- he mmum enroy! The m-enroy rnle o rmeerzon oers dul erseve o he MLE. Er CMU