Exponential Families and Friends: Learning the Parameters of the a Fully Observed BN. Kayhan Batmanghelich

Transcription

1 Epoetia Famiies ad Frieds: Learig te Parameters of te a Fuy Observed BN Kaya Batmageic

2 Macie Learig Te data ispires te structures we wat to predict Iferece fids {best structure, margias, partitio fuctio} for a ew observatio Iferece is usuay caed as a subroutie i earig Domai Kowedge Combiatoria Optimizatio ML Matematica Modeig Optimizatio Our mode defies a score for eac structure It aso tes us wat to optimize Learig tues te parameters of te mode 2

3 Macie Learig Data Mode X time fies ike a arrow X2 X 3 X 4 X 5 Objective Iferece Learig Iferece is usuay caed as a subroutie i earig 3

4 Today s Lecture X2 X X 3 X 4 X 5 px,x 2,X 3,X 4,X 5 px 5 X 3 px 4 X 2,X 3 px 3 px 2 X px 4

5 Today s Lecture X2 X X 3 X 4 X 5 px,x 2,X 3,X 4,X 5 px 5 X 3 px 4 X 2,X 3 px 3 px 2 X px 5

6 Today s Lecture X2 X X 3 X 4 X 5 px,x 2,X 3,X 4,X 5 px 5 X 3 px 4 X 2,X 3 px 3 px 2 X px How do we defie ad ear tese coditioa ad margia distributios for a Bayes Net? 6

7 Today s Lecture. Epoetia Famiy Distributios A cadidate for margia distributios, px i 2. Geeraized Liear Modes Coveiet form for coditioa distributios, px j X i 3. Learig Fuy Observed Bayes Nets Easy taks to decomposabiity 7

8 A cadidate for margia distributios, px i. EXPONENTIAL FAMILY 8

9 Wy te Epoetia Famiy?. Pitma-Koopma-Darmois teorem: it is te oy famiy of distributios wit sufficiet statistics tat do ot grow wit te size of te dataset 2. Oy famiy of distributios for wic cojugate priors eist see Murpy tetbook for a descriptio 3. It is te distributio tat is cosest to uiform i.e. maimizes etropy subject to momet matcig costraits 4. Key to Geeraized Liear Modes et sectio 5. Icudes some of your favorite distributios! Adapted from Murpy 202 tetbook 9

10 Witeboard Defiitio of mutivariate epoetia famiy 0

11 Witeboard Eampe : Categorica distributio

12 Witeboard Eampe 2: Mutivariate Gaussia distributio 2

13 Momets ad te Partitio Fuctio p; ep T A 3

14 Momets ad te Partitio Fuctio p; ep T T A r A E[T ] r 2 A E[T T T ] E[T ]E[T ] T 4

15 Sufficiecy For!#; %, T is sufficiet for q if tere is o iformatio i X regardig q beyod tat i T. We ca trow away X for te purpose of iferece w.r.t. q. Bayesia view X T p q T, p q T q Frequetist view X T p T, q p T q Te Neyma factorizatio teorem T is sufficiet for q if X T q p y 2, T, q y T, q, T Þ p q g T, q, T Eric CMU,

16 Sufficiecy p; ep T T A Let s assume i iid p,, ; p; Y j A ep T X T j A A j j 6

17 MLE for Epoetia Famiy For iid data, te og-ikeiood is Take derivatives ad set to zero: Tis amouts to momet matcig. We ca ifer te caoica parameters usig Eric CMU, { } å å Õ - ø ö ç è æ + - T T NA T A T D og ep og ; å å å Þ - MLE T N T N A A N T 0 µ! MLE MLE µ y!!

18 Eampes Gaussia: Mutiomia: Poisso: Eric CMU, [ ] [ ] / og ; vec vec ; k T T A T S + S S - S p µ µ µ å å Þ MLE N T N µ [ ] 0 ø ö ç è æ ø ö ç è æ - - ú û ù ê ë é ø ö ç è æ å å - ; e A T K k K k k K k k p p p å Þ MLE N µ! og e A T å Þ MLE N µ

19 Witeboard Bayesia estimatio of epoetia famiy p; ep T T A We ave observed iid sampes ad we are iterested i p {,, {z } D 25

20 Posterior Mea Uder Cojugate Prior p; ep T T p ;, 0 ep A T 0 A Ã, 0 p D p ; + X i T i ; + 0 Posterior mea of! E[ D] + 0 P i T i

21 Coveiet form for coditioa distributios, px j X i 2. GENERALIZED LINEAR MODELS 27

22 Wy Geeraized Liear Modes? GLIMs. Geeraizatio of iear regressio, ogistic regressio, probit regressio, etc. 2. Provides a framework for creatig ew coditioa distributios tat come wit some coveiet properties 3. Specia case: GLMs wit caoica respose fuctios are easy to trai wit MLE. 4. No Free Luc: Wat about Bayesia estimatio of GLMs? Ufortuatey, we ave to tur to approimatio teciques sice, i geera, tere is't a cosed form of te posterior. 28

23 Geeraized Liear Modes GLMs GLM Te observed iput is assumed to eter ito te mode T via a iear combiatio of its eemets q Te coditioa mea µ is represeted as a fuctio f of, were f is kow as te respose fuctio Te observed output y is assumed to be caracterized by a epoetia famiy distributio wit coditioa mea µ. X Y N Eric CMU,

24 Witeboard Costructive defiitio of GLMs Defiitio of GLMs wit caoica respose fuctios 30

25 Eampes of te caoica respose fuctios 3

26 Witeboard MLE wit GLM wit Caoica respose 32

27 MLE for GLMs wit caoica respose Log-ikeiood Lw X i og y i + X i Derivative of Log-ikeiood y i w T i A i r w Lw X i i y i da i d i d i é- - ê ê - - X ê! ê ë- - é y ù ê ú ê y y " 2 ú ê! ú ê ú ëy û! 2 - -ù - - ú ú! ú ú - -û X i y i µ i i Oie earig for caoica GLMs X T y µ Tis is a fuctio of! Stocastic gradiet ascet east mea squares LMS agoritm: w t+ w t + y i µ t i i Step egt 33

28 Batc earig for caoica GLMs Te Hessia matri é- - ê ê - - X ê! ê ë- - é y ù ê ú ê y y " 2 ú ê! ú ê ú ëy û! 2 - -ù - - ú ú! ú ú - -û Ivoves te secod derivative of!# 34

29 Iterativey Reweigted Least Squares IRLS r w Lw X T y µ Reca Newto-Rapso metods wit cost fuctio w t+ w t + H w t rlw t X T Sw t X X T Sw t Xw t + X T y µ X T Sw t X X T Sw t z t z t Xw t + Sw t y µ t 35

30 Iterativey Reweigted Least Squares IRLS r w Lw X T y µ Reca Newto-Rapso metods wit cost fuctio w t+ w t + H w t rlw t X T Sw t X X T Sw t Xw t + X T y µ X T Sw t X X T Sw t z t z t Xw t + Sw t y µ t It ooks ike! "! #$! " % 36

31 Iterativey Reweigted Least Squares IRLS r w Lw X T y µ Reca Newto-Rapso metods wit cost fuctio w t+ w t + H w t rlw t X T Sw t X X T Sw t Xw t + X T y µ X T Sw t X X T Sw t z t z t Xw t + Sw t y µ t Tis ca be uderstood as sovig te foowig " Iterativey reweigted east squares " probem 37

32 Eampes r w Lw X T y µ Reca Newto-Rapso metods wit cost fuctio w t+ w t + H w t rlw t X T Sw t X X T Sw t Xw t + X T y µ X T Sw t X X T Sw t z t z t Xw t + Sw t y µ t 38

33 Practica Issues 39 It is very commo to use reguarized maimum ikeiood. IRLS takes!#$ % per iteratio, were N umber of traiig cases ad d dimesio of iput. Quasi-Newto metods, tat approimate te Hessia, work faster. Cojugate gradiet takes!#$ per iteratio, ad usuay works best i practice. Stocastic gradiet descet ca aso be used if N is arge c.f. perceptro rue. Eric CMU, q q q s q q q s q q T T T y y I p y e y p T ± å - - og, Norma ~,

34 Today s Lecture. Epoetia Famiy Distributios A cadidate for margia distributios, px i 2. Geeraized Liear Modes Coveiet form for coditioa distributios, px j X i 3. Learig Fuy Observed Bayes Nets Easy taks to decomposabiity 50

35 Easy taks to decomposabiity 3. LEARNING FULLY OBSERVED BNS 5

36 Simpe GMs are te buidig bocks of compe BNs Desity estimatio Parametric ad oparametric metods Regressio Liear, coditioa miture, oparametric Cassificatio Geerative ad discrimiative approac X X Q X µ,s X Y Q X Eric CMU,

37 Decomposabe ikeiood of a BN Reca from Lecture 2 Cosider te distributio defied by te directed acycic GM: p q p q p 2, q2 p 3, q3 p 4 2, 3, q4 Tis is eacty ike earig four separate sma BNs, eac of wic cosists of a ode ad its parets. X X X X X 2 X 3 X 2 X 2 X 3 X 3 X 4 X 4 Eric CMU,

38 Learig Fuy Observed BNs X2 X X 3 argma argma og px,x 2,X 3,X 4,X 5 og px 5 X 3, 5 + og px 4 X 2,X 3, 4 + og px og px 2 X, 2 + og px X 4 X 5 argma og px 2 argma 2 og px 2 X, 2 3 argma 3 og px argma 4 og px 4 X 2,X 3, 4 5 argma 5 og px 5 X 3, 5 54

39 Summary. Epoetia Famiy Distributios A cadidate for margia distributios, px i Eampes: Mutiomia, Diricet, Gaussia, Gamma, Poisso MLE as cosed form soutio Bayesia estimatio easy wit cojugate priors Sufficiet statistics by ispectio 2. Geeraized Liear Modes Coveiet form for coditioa distributios, px j X i Specia case: GLIMs wit caoica respose Output y foows a epoetia famiy Iput itroduced via a iear combiatio MLE for GLIMs wit caoica respose by SGD I geera, Bayesia estimatio reies o approimatios 3. Learig Fuy Observed Bayes Nets Easy taks to decomposabiity 55