Physical Fluctuomatics 4th Maximum likelihood estimation and EM algorithm

Transcription

1 hyscal Fluctuomatcs 4th Maxmum lkelhood estmaton and EM alorthm Kazuyuk Tanaka Graduate School o Inormaton Scences Tohoku Unversty kazu@smapp.s.tohoku.ac.jp hscal Fluctuomatcs (Tohoku Unversty)

2 Textbooks Kazuyuk Tanaka: Introducton o Imae rocessn by robablstc Models Morkta ublshn Co. Ltd. 6 (n Japanese) Chapter 4. hscal Fluctuomatcs (Tohoku Unversty)

3 Statstcal Machne Learnn and Model Selecton Inerence o robablstc Model by usn Model Selecton Statstcal Machne Learnn Maxmum Lkelhood Complete and Incomplete data Extenson EM alorthm Implement by Bele ropaaton and Markov Chan Monte Carlo method Akake Inormaton Crtera Akake Bayes Inormaton Crtera etc. hscal Fluctuomatcs (Tohoku Unversty) 3

4 Maxmum Lkelhood Estmaton arameter µ hscal Fluctuomatcs (Tohoku Unversty) 4

5 Maxmum Lkelhood Estmaton arameter µ N N µ π ( µ ) exp ( ) { } V N hscal Fluctuomatcs (Tohoku Unversty) 5

6 Maxmum Lkelhood Estmaton arameter µ N N µ π ( µ ) exp ( ) { } V N hscal Fluctuomatcs (Tohoku Unversty) 6

7 Maxmum Lkelhood Estmaton arameter µ N N µ π ( µ ) exp ( ) { } V N Hstoram hscal Fluctuomatcs (Tohoku Unversty) 7

8 Maxmum Lkelhood Estmaton arameter µ N N µ π ( µ ) exp ( ) { } V N Hstoram hscal Fluctuomatcs (Tohoku Unversty) 8

9 Maxmum Lkelhood Estmaton arameter µ { } V N N N µ π ( µ ) exp ( ) ( ˆ µ ˆ ) ar max ( µ ) µ Hstoram hscal Fluctuomatcs (Tohoku Unversty) 9

10 Maxmum Lkelhood Estmaton arameter µ N N µ π ( µ ) exp ( ) ( ˆ µ ˆ ) ar max ( µ ) µ robablty densty uncton or data wth averae µ and varance s rearded as lkelhood uncton or averae µ and varance when data s ven. hscal Fluctuomatcs (Tohoku Unversty)

11 Maxmum Lkelhood Estmaton arameter µ N ( ˆ µ ˆ ) ar max N ( µ ) µ µ π ( µ ) exp ( ) Extremum Condton ( µ ) µ ( µ ) µ ˆ µ ˆ µ ˆ µ ˆ robablty densty uncton or data wth averae µ and varance s rearded as lkelhood uncton or averae µ and varance when data s ven. hscal Fluctuomatcs (Tohoku Unversty)

12 Maxmum Lkelhood Estmaton arameter µ N ( ˆ µ ˆ ) ar max N ( µ ) µ µ π ( µ ) exp ( ) Extremum Condton ( µ ) µ ( µ ) Sample Averae µ ˆ µ ˆ µ ˆ µ ˆ robablty densty uncton or data wth averae µ and varance s rearded as lkelhood uncton or averae µ and varance when data s ven. N N ˆ µ ˆ ( ˆ µ ) N N hscal Fluctuomatcs (Tohoku Unversty) Sample Devaton

13 Maxmum Lkelhood Estmaton arameter µ N ( ˆ µ ˆ ) ar max N ( µ ) µ µ π ( µ ) exp ( ) Extremum Condton Hstoram ( µ ) µ ( µ ) µ ˆ µ ˆ µ ˆ µ ˆ Sample Averae robablty densty uncton or data wth averae µ and varance s rearded as lkelhood uncton or averae µ and varance when data s ven. N N ˆ µ ˆ ( ˆ µ ) N N hscal Fluctuomatcs (Tohoku Unversty) 3 Sample Devaton

14 Maxmum Lkelhood Hyperparameter arameter N Extremum Condton Bayes Formula ( ) ( ) ( ) ( ) ( ) N ˆ s unknown ˆ ˆ ar max + N N exp ( ) N Marnal Lkelhood ( ) ( ) ˆ π hscal Fluctuomatcs (Tohoku Unversty) 4 N exp π ( ) ˆ d

15 robablstc Model or Snal rocessn Observable Source Snal + Whte Gaussan Nose Nose Transmsson Source Snal Observable osteror robablty ' & % r { Source Snal Observable } Bayes Formula Lkelhood ror robablty ' & %' &% r { Observable Source Snal} r{ Source Snal} r{ ObservableSnal} $ # Marnal Lkelhood hscal Fluctuomatcs (Tohoku Unversty) 5

16 ror robablty or Source Snal ( ) exp ( ) Z ror Z ror { j} E exp One dmensonal j { j} E j { j} E j E:Set o all the lnks Imae X 3 X 3 4 X hscal Fluctuomatcs (Tohoku Unversty) 6

17 Generatn rocess Addtve Whte Gaussan Nose ( ) V exp ( ) π ( ~ N V Set o all the nodes hscal Fluctuomatcs (Tohoku Unversty) ) 7

18 hscal Fluctuomatcs (Tohoku Unversty) 8 robablstc Model or Snal rocessn V exp π E j j Z } { pror exp N データ N Hyperparameter arameter d d ˆ osteror robablty

19 hscal Fluctuomatcs (Tohoku Unversty) 9 N Maxmum Lkelhood n Snal rocessn ar max ˆ ˆ ˆ ˆ ˆ ˆ Extremum Condton d N Hyperparameter arameter Incomplete Marnal Lkelhood

20 hscal Fluctuomatcs (Tohoku Unversty) N Maxmum Lkelhood and EM alorthm ˆ ˆ ˆ ˆ Extemum Condton d N Hyperparameter arameter d Q ʹ ʹ ʹ ʹ ln ) ( ) ( ar max ) ( ) ( M Step : Update ) ( ) ( E Step :Calculate ) ( t t Q t t t t Q + + ʹ ʹ ʹ ʹ ʹ ʹ ʹ ʹ Q Q Expectaton Maxmzaton (EM) alorthm provde us one o Extremum onts o Marnal Lkelhood. Q uncton Marnal Lkelhood Incomplete

21 Ornal Snal Model Selecton n One Dmensonal Snal Deraded Snal 4 Estmated Snal ˆ Expectaton Maxmzaton (EM) Alorthm.4.3 (t) (). () hscal Fluctuomatcs (Tohoku Unversty)

22 Model Selecton n Nose Reducton Ornal Imae 4 Deraded Imae Estmate o Ornal Imae MSE ˆ ˆ EM alorthm and Bele ropaaton (). () MSE Ω Ω ( ˆ ) MSE hscal Fluctuomatcs (Tohoku Unversty) ˆ ˆ

23 Summary Maxmum Lkelhood and EM alorthm Statstcal Inerence by Gaussan Graphcal Model hscal Fluctuomatcs (Tohoku Unversty) 3

24 ractce 3- Let us suppose that data {...N-} are enerated by accordn to the ollown probablty densty uncton: N µ exp ( ) µ π rove that estmates or averae µ and varance o the maxmum lkelhood ˆ µ ˆ ar max µ are ven as N ( µ ) N ˆ µ ˆ ( ˆ) N µ N where N hscal Fluctuomatcs (Tohoku Unversty) 4