Density estimation. Density estimations. CS 2750 Machine Learning. Lecture 5. Milos Hauskrecht 5329 Sennott Square

Transcription

1 Lecure 5 esy esmao Mlos Hauskrec mlos@cs..edu 539 Seo Square esy esmaos ocs: esy esmao: Mamum lkelood ML Bayesa arameer esmaes M Beroull dsrbuo. Bomal dsrbuo Mulomal dsrbuo Normal dsrbuo Eoeal famly Noaramerc famly

2 aramerc desy esmao aramerc desy esmao: se of radom varables X { X X X d} model of e dsrbuo over varables X w arameers : ˆ X aa.. } { Objecve: fd arameers suc a X descrbes daa e bes arameer esmao learg Mamum lkelood ML arg ma ML Mamum a oseror robably M M Bayesa arameer esmao use e oseror desy Eeced value arg ma EX d

3 3 Eoeal famly of dsrbuo Eoeal famly of dsrbuos well beaved dsrbuos w resec o ML ad Bayesa udag Cojugae coces for some of e dsrbuos from e eoeal famly: Bomal Bea Mulomal - rcle Eoeal Gamma osso Iverse Gamma Gaussa - Gaussa mea ad Wsar covarace Sequeal Bayesa arameer esmao Sequeal Bayesa aroac Uder e d e esmaes of e oseror ca be comued cremeally for a sequece of daa os If we use a cojugae ror we ge back e same oseror ssume we sl e daa e las eleme ad e res e: d d ew ror

4 Eoeal famly Eoeal famly: all robably mass / desy fucos a ca be wre e eoeal ormal form f e[ ] a vecor of aural or caocal arameers a fuco referred o as a suffce sasc a fuco of s less mora a ormalzao cosa a aro fuco e{ } d Oer commo form: [ ] f e Eoeal famly: eamles Beroull dsrbuo π π π π e + π π π e{ π } e π Eoeal famly f e[ ] arameers???? 4

5 Eoeal famly: eamles Beroull dsrbuo π π π π e + π π π e{ π } e π Eoeal famly f e[ ] arameers π π π + e Eoeal famly arameers Eoeal famly: eamles Uvarae Gaussa dsrbuo µ e[ µ ] π µ µ e e π???? [ ] f e 5

6 6 Eoeal famly: eamles Uvarae Gaussa dsrbuo Eoeal famly arameers e e µ µ π / / µ π / + 4 e e µ [ ] e f ] e[ µ π µ Eoeal famly For d samles e lkelood of daa s Imora: e dmesoaly of e suffce sasc remas e same for dffere samle szes a s dffere umber of eamles [ ] e e e

7 7 Eoeal famly e lkelood of daa s Omzg e lkelood For e ML esmae mus old e l + 0 l Eoeal famly Rewrg e grade: Resul: For e ML esmae e arameers sould be adjused suc a e eecao of e sasc s equal o e observed samle sascs { } d e { } { } d d e e { } d e E E

8 Momes of e dsrbuo For e eoeal famly e k- mome of e sasc corresods o e k- dervave of If s a comoe of e we ge e momes of e dsrbuo by dffereag s corresodg aural arameer Eamle: Beroull π π e + π π π + e ervaves: e + e π + e + e π π + e Eoeal famly of dsrbuo Bayesa arameer esmae We ave see cojugae coces for some of e dsrbuos from e eoeal famly: Bomal Bea Mulomal - rcle Eoeal Gamma osso Iverse Gamma Gaussa - Gaussa mea ad Wsar covarace 8

9 9 Cojugae rors For ay member of e eoeal famly ere ess a ror: Suc a for eamles e oseror s Noe a: [ ] f e e [ ] g u e + + e g e Cojugae rors For ay member of e eoeal famly ere ess a ror: Suc a for eamles e oseror s Noe a: [ ] f e [ ] g u e + + e g seudo-observaos

10 Noaramerc Meods aramerc dsrbuo models are: resrced o secfc forms wc may o always be suable; Eamle: modellg a mulmodal dsrbuo w a sgle umodal model. Noaramerc aroaces: make few assumos abou e overall sae of e dsrbuo beg modelled. Noaramerc Meods Hsogram meods: aro e daa sace o dsc bs w wds ad cou e umber of observaos eac b. NΔ Ofe e same wd s used for all bs. acs as a smoog arameer. I a -dmesoal sace usg M bs eac dme-so wll requre M bs! 0

11 Noaramerc Meods ssume observaos draw from a desy ad cosder a small rego R coag suc a d R e robably a K ou of N observaos le sde R s BKN ad f N s large K N If e volume of R V s suffcely small s aromaely cosa over R ad us V V K NV Noaramerc Meods: kerel meods Kerel esy Esmao: F V esmae K from e daa. Le R be a yercube cered o ad defe e kerel fuco arze wdow k I follows a ad ece / / 0 K N oerwse k N N k

12 Noaramerc Meods: smoo kerels o avod dscoues because of sar boudares use a smoo kerel e.g. a Gaussa y kerel suc a acs as a smooer. wll work. Noaramerc Meods: knn esmao Neares Negbour esy Esmao: f K esmae V from e daa. Cosder a yer-sere cered o ad le grow o a volume V* a cludes K of e gve N daa os. e K acs as a smooer