Generalized Linear Models + Learning Fully Observed Bayes Nets

Transcription

1 School of Compuer Sciece Probabilisic Graphical Moels Geeralize Liear Moels + Learig Full Observe Baes Nes Reaigs: KF Chap. 7 Jora Chap. 8 Jora Chap Ma Gormle Lecure 5 Jauar 7, 06

2 Machie Learig he aa ispires he srucures we wa o preic Iferece fis {bes srucure, margials, pariio fucio} for a ew observaio Iferece is usuall calle as a subrouie i learig Domai Kowlege Combiaorial Opimizaio ML Mahemaical Moelig Opimizaio Our moel efies a score for each srucure I also ells us wha o opimize Learig ues he parameers of he moel

3 3 Al Machie Learig Alice saw Bob o a hill wih a elescope Daa 4 ime flies lie a arrow Moel ime flies lie a arrow 3 ime flies lie a arrow ime flies lie a arrow Objecive 4 5 Iferece ime flies lie a arrow Learig Iferece is usuall calle as a subrouie i learig 3

4 oa s Lecure p,, 3, 4, 5 p 5 3 p 4, 3 p 3 p p 4

5 oa s Lecure p,, 3, 4, 5 p 5 3 p 4, 3 p 3 p p 5

6 oa s Lecure p,, 3, 4, 5 p 5 3 p 4, 3 p 3 p p How o we efie a lear hese coiioal a margial isribuios for a Baes Ne? 6

7 oa s Lecure. Epoeial Famil Disribuios A caiae for margial isribuios, p i. Geeralize Liear Moels Coveie form for coiioal isribuios, p j i 3. Learig Full Observe Baes Nes Eas has o ecomposabili 7

8 A caiae for margial isribuios, p i. EPONENIAL FAMILY 8

9 Wh he Epoeial Famil?. Pima- Koopma- Darmois heorem: i is he ol famil of isribuios wih sufficie saisics ha o o grow wih he size of he aase. Ol famil of isribuios for which cojugae priors eis see Murph eboo for a escripio 3. I is he isribuio ha is closes o uiform i.e. maimizes erop subjec o mome machig cosrais 4. Ke o Geeralize Liear Moels e secio 5. Iclues some of our favorie isribuios Aape from Murph 0 eboo 9

10 Whieboar Defiiio of mulivariae epoeial famil Eample : Caegorical isribuio Eample : Dirichle isribuio 0

11 Epoeial famil, a basic builig bloc For a umeric raom variable p h ep h ep Z { A } { } is a epoeial famil isribuio wih aural caoical parameer Fucio is a sufficie saisic. Fucio A log Z is he log ormalizer. Eamples: Beroulli, muliomial, Gaussia, Poisso, gamma,... Eric CMU,

12 Eample: Mulivariae Gaussia Disribuio For a coiuous vecor raom variable R : Epoeial famil represeaio Noe: a -imesioal Gaussia is a + -parameer isribuio wih a + - eleme vecor of sufficie saisics bu because of smmer a posiivi, parameers are cosraie a have lower egree of freeom Eric CMU, { } Σ Σ Σ + Σ Σ Σ Σ log r ep ep, / / / π π p [ ] [ ] [ ] / log r log ; vec a, vec, vec ; h A Σ + Σ Σ Σ Σ Σ π Mome parameer Naural parameer

13 Eample: Muliomial isribuio For a biar vecor raom variable Epoeial famil represeaio Eric CMU, , muli ~ π K p K π π π π π l ep [ ] 0 l l ; l h e A K K K π π π + + l l ep l l ep K K K K K K K π π π π π

14 Cumula Geeraig Proper Firs cumula aa. Mea Seco cumula aa. Variace or Firs ceral mome Eric CMU, { } { } [ ] ep ep log E Z h h Z Z Z Z A { } { } [ ] [ ] [ ] ep ep Var E E Z Z Z h Z h A

15 Mome esimaio We ca easil compue momes of a epoeial famil isribuio b aig he erivaives of he log ormalizer A. he q h erivaive gives he q h ceere mome. A mea A variace Whe he sufficie saisic is a sace vecor, parial erivaives ee o be cosiere. Eric CMU,

16 Mome vs caoical parameers he mome parameer ca be erive from he aural caoical parameer A is cove sice A A ef [ ] E [ ] > 0 Var Hece we ca iver he relaioship a ifer he caoical parameer from he mome parameer -o-: ef ψ A 8 4 * A isribuio i he epoeial famil ca be parameerize o ol b - he caoical parameerizaio, bu also b - he mome parameerizaio. Eric CMU,

17 MLE for Epoeial Famil For ii aa, he log-lielihoo is ae erivaives a se o zero: his amous o mome machig. We ca ifer he caoical parameers usig Eric CMU, { } + NA h A h D log ep log ; l MLE N N A A N 0 l MLE MLE ψ

18 Eamples Gaussia: Muliomial: Poisso: Eric CMU, [ ] [ ] / log ; vec vec ; h A Σ + Σ Σ Σ π MLE N N [ ] 0 l l ; l h e A K K K π π π MLE N log h e A λ λ MLE N

19 Sufficiec For p q, is sufficie for if here is o iformaio i regarig beo ha i. We ca hrow awa for he purpose of iferece w.r... Baesia view p, p Frequeis view p, p he Nema facorizaio heorem is sufficie for if p ψ,, ψ,, p g, h, Eric CMU,

20 Whieboar Baesia esimaio of epoeial famil 0

21 Coveie form for coiioal isribuios, p j i. GENERALIZED LINEAR MODELS

22 Wh Geeralize Liear Moels? GLIMs. Geeralizaio of liear regressio, logisic regressio, probi regressio, ec.. Provies a framewor for creaig ew coiioal isribuios ha come wih some coveie properies 3. Special case: GLIMs wih caoical respose fucios are eas o rai wih MLE. 4. No Free Luch: Wha abou Baesia esimaio of GLIMs? Uforuael, we have o ur o approimaio echiques sice, i geeral, here is' a close form of he poserior.

23 Geeralize Liear Moels GLIMs GLIM he observe ipu is assume o eer io he moel via a liear combiaio of is elemes ξ he coiioal mea is represee as a fucio fξ of ξ, where f is ow as he respose fucio he observe oupu is assume o be characerize b a epoeial famil isribuio wih coiioal mea. Y N Eric CMU,

24 Whieboar Cosrucive efiiio of GLIMs Defiiio of GLIMs wih caoical respose fucios 4

25 Geeralize Liear Moels GLIMs he graphical moel Liear regressio Discrimiaive liear classificaio Commoali: moel E p Yf Wha is p? he co. is. of Y. Wha is f? he respose fucio. Y N Eric CMU,

26 GLIM, co. ξ f ψ EP { A } p h ep { } p, φ h, φep φ A he choice of ep famil is cosraie b he aure of he aa Y Eample: is a coiuous vecor à mulivariae Gaussia is a class label à Beroulli or muliomial he choice of he respose fucio Followig some mil cosrais, e.g., [0,]. Posiivi Caoical respose fucio: f ψ I his case irecl correspos o caoical parameer. Eric CMU,

27 Eample caoical respose fucios Eric CMU,

28 MLE for GLIMs wih aural respose Log-lielihoo Derivaive of Log-lielihoo Olie learig for caoical GLIMs Sochasic graie asce leas mea squares LMS algorihm: Eric CMU, A h log l A l his is a fie poi fucio because is a fucio of ρ + + size is a sep a where ρ

29 Bach learig for caoical GLIMs he Hessia mari where is he esig mari a which ca be compue b calculaig he erivaive of A Eric CMU, W H sice l [ ] N N W,, iag "

30 Recall LMS Cos fucio i mari form: o miimize J, ae erivaive a se o zero: Eric CMU, J i i i " J r r r r he ormal equaios *

31 Ieraivel Reweighe Leas Squares IRLS Recall Newo-Raphso mehos wih cos fucio J We ow have Now: where he ajuse respose is his ca be uersoo as solvig he followig " Ieraivel reweighe leas squares " problem Eric CMU, J H + W H J [ ] z W W W W H l W z + mi arg z W z + *

32 Eample : logisic regressio sigmoi classifier he coiio isribuio: a Beroulli where is a logisic fucio p is a epoeial famil fucio, wih mea: a caoical respose fucio IRLS Eric CMU, p e + [ ] e E + ξ N N W

33 Logisic regressio: pracical issues I is ver commo o use regularize maimum lielihoo. IRLS aes ON 3 per ieraio, where N umber of raiig cases a imesio of ipu. Quasi-Newo mehos, ha approimae he Hessia, wor faser. Cojugae graie aes ON per ieraio, a usuall wors bes i pracice. Sochasic graie esce ca also be use if N is large c.f. percepro rule: Eric CMU, λ σ λ σ l I p e p 0 + ± log, Normal ~, λ σ l

34 Eample : liear regressio he coiio isribuio: a Gaussia where is a liear fucio p is a epoeial famil fucio, wih mea: a caoical respose fucio IRLS Eric CMU, [ ] E ξ I W { } ep ep,, / / π A h p Σ Σ Σ Σ z W W Y Seepes esce Normal equaio Rescale

35 oa s Lecure. Epoeial Famil Disribuios A caiae for margial isribuios, p i. Geeralize Liear Moels Coveie form for coiioal isribuios, p j i 3. Learig Full Observe Baes Nes Eas has o ecomposabili 35

36 Eas has o ecomposabili 3. LEARNING FULLY OBSERVED BNS 36

37 Simple GMs are he builig blocs of comple BNs Desi esimaio Parameric a oparameric mehos Regressio Liear, coiioal miure, oparameric Classificaio Geeraive a iscrimiaive approach Q,σ Y Q Eric CMU,

38 Decomposable lielihoo of a BN Cosier he isribuio efie b he irece acclic GM: p p p, p 3, 3 p 4, 3, 4 his is eacl lie learig four separae small BNs, each of which cosiss of a oe a is pares Eric CMU,

39 Learig Full Observe BNs 3 argma argma log p,, 3, 4, 5 log p 5 3, 5 + log p 4, 3, 4 + log p log p, + log p 4 5 argma log p argma log p, 3 argma 3 log p argma 4 log p 4, 3, 4 5 argma 5 log p 5 3, 5 39

40 Summar. Epoeial Famil Disribuios A caiae for margial isribuios, p i Eamples: Muliomial, Dirichle, Gaussia, Gamma, Poisso MLE has close form soluio Baesia esimaio eas wih cojugae priors Sufficie saisics b ispecio. Geeralize Liear Moels Coveie form for coiioal isribuios, p j i Special case: GLIMs wih caoical respose Oupu follows a epoeial famil Ipu irouce via a liear combiaio MLE for GLIMs wih caoical respose b SGD I geeral, Baesia esimaio relies o approimaios 3. Learig Full Observe Baes Nes Eas has o ecomposabili 40