Maximum Likelihood Estimation

Transcription

1 Mau Lkelhood aon Beln Chen Depaen of Copue Scence & Infoaon ngneeng aonal Tawan oal Unvey Refeence:. he Alpaydn, Inoducon o Machne Leanng, Chape 4, MIT Pe, 4

2 Saple Sac and Populaon Paaee A Scheac Depcon Populaon (aple pace) Saple Paaee Infeence Sac SP - Beln Chen

3 Sac Inoducon Any value (o funcon) ha calculaed fo a gven aple Sacal nfeence: ake a decon ung he nfoaon povded by a aple (o a e of eaple/nance) Paaec ehod Aue ha eaple ae dawn fo oe dbuon ha obey a known odel p ( ) Advanage: he odel well defned up o a all nube of paaee.g., ean and vaance ae uffcen ac fo he Gauan dbuon Model paaee ae ypcally eaed by ehe au lkelhood eaon o Bayean (MAP) eaon SP - Beln Chen 3

4 Mau Lkelhood aon (ML) (/) { } Aue he nance,, K,, K, ae ndependen and dencally dbued (d), and dawn fo oe known pobably dbuon ~ p ( θ ) θ : odel paaee (aued o be fed bu unknown hee) θ ML aep o fnd ha ake he o lkely o be dawn aely, aze he lkelhood of he nance, K, ae d l ( θ ) p ( θ ) p (, L, θ ) p ( θ ) SP - Beln Chen 4

5 ML (/) Becaue ah wll no change he value of when ake au (onooncally nceang/deceang) Fndng θ ha aze he lkelhood of he nance equvalen o fndng θ ha aze he lkelhood of he aple L ( θ ) l ( θ ) p ( θ ) θ a b a b A we hall ee, ahc opeaon can fuhe plfy he copuaon when eang he paaee of hoe dbuon ha have eponen SP - Beln Chen 5

6 ML: Benoull Dbuon (/3) Benoull Dbuon A ando vaable ake ehe he value (wh pobably ) o he value (wh pobably ) Can be hough of a geneaed fo wo dnc ae The aocaed pobably dbuon ( ), {, } P The lkelhood fo a e of d nance dawn fo Benoull dbuon,, K,, K, θ L ( ) ( ) { } ( ) SP - Beln Chen 6

7 ML: Benoull Dbuon (/3) ML of he dbuon paaee ˆ The eae fo he ao of he nube of occuence of he even ( ) o he nube of epeen The epeced value fo [ ] {,} P ( ) ( ) The vaance value fo va ( ) [ ] [ ] ( ) SP - Beln Chen 7

8 ML: Benoull Dbuon (3/3) Append A dl d d y dy y d ˆ y y The au lkelhood eae of he ean he aple aveage SP - Beln Chen 8

9 ML: Mulnoal Dbuon (/4) Mulnoal Dbuon A genealzaon of Benoull dbuon { } The value of a ando vaable can be one of K uually ecluve and ehauve ae,,, K wh L pobable,, L, K, epecvely The aocaed pobably dbuon p K, f chooe ae ohewe K The lkelhood fo a e of d nance dawn fo a ulnoal dbuon L ( ) K {,, K,, K, } SP - Beln Chen 9

10 ML: Mulnoal Dbuon (/4) ML of he dbuon paaee ˆ The eae fo he ao of he nube of epeen wh oucoe of ae o he nube of epeen SP - Beln Chen

11 ML: Mulnoal Dbuon (3/4) Append B K K K L K K : conan wh, λ L g Lagange Mulple λ K K λ λ λ SP - Beln Chen ˆ Lagange Mulple: hp://

12 ML: Mulnoal Dbuon (4/4) P(B)3/ P(W)4/ P(R)3/ SP - Beln Chen

13 ML: Gauan Dbuon (/3) Alo called oal Dbuon Chaacezed wh ean and vaance ( ) p( ) ep, - < π < Recall ha ean and vaance ae uffcen ac fo Gauan The lkelhood fo a e of d nance dawn fo Gauan dbuon ( ),, K,, K, L (, ) e π ( π ) ( ) { } SP - Beln Chen 3

14 ML: Gauan Dbuon (/3) ML of he dbuon paaee and ˆ ( ) aple aveage ˆ ˆ aple vaance Rend ha and ae ll fed bu unknown SP - Beln Chen 4

15 ML: Gauan Dbuon (3/3) Append C, π L ˆ, L ˆ ˆ, L SP - Beln Chen 5

16 valuang an ao : Ba and Vaance (/6) The ean quae eo of he eao d can be fuhe decopoed no wo pa epecvely copoed of ba and vaance [ ] ( d, θ ) ( d θ ) [( [ ] [ ] ) ] d d d θ [( [ ]) ( [ ] ) ( [ ])( [ ] )] d d d θ d d d θ ( d [ d ] ) ( d θ ) [ ( d d ) ( [ d ] θ ) ] [ ] [ [ ] ] [ ] conan conan [ ( [ ]) ] d d ( [ d ] θ ) [ ( d [ d ] ) ] ( [ d ] θ ) [ ( [ ]) ] d d ( [ d ] θ ) vaance ba SP - Beln Chen 6

17 valuang an ao : Ba and Vaance (/6) SP - Beln Chen 7

18 valuang an ao : Ba and Vaance (3/6) aple : aple aveage and aple vaance { } Aue aple,, K,, K, ae ndependen and dencally dbued (d), and dawn fo oe known pobably dbuon wh ean and vaance Mean [ ] p Vaance ( ) [ ] [ ] [ ] Saple aveage (ean) fo he obeved aple Saple vaance fo he obeved aple o ( )? SP - Beln Chen 8

19 valuang an ao : Ba and Vaance (4/6) aple (coun.) Saple aveage an unbaed eao of he ean [ ] [ ] [ ] Va( ) a alo a conen eao: Va Va ( ) Va( ) Va Va ( a b) a Va( ) ( Y ) Va( ) Va( Y ) SP - Beln Chen 9

20 valuang an ao : Ba and Vaance (5/6) aple (coun.) Saple vaance an aypocally unbaed eao of Saple vaance an aypocally unbaed eao of he vaance [ ]..d. ae ' [ ] [ ] SP - Beln Chen

21 valuang an ao : Ba and Vaance (6/6) aple (coun.) Saple vaance an aypocally unbaed eao of he vaance ( ) [ ] [ ] [ ] [ ] Va [ ] [ ] [ ] Va ( ) ( ) ( ) [ ] [ ] [ ] ( [ ] ) The ze of he obeved aple e SP - Beln Chen

22 Ba and Vaance: aple dffeen aple fo an unknown populaon ( y), y F 3 y F ε eo of eaueen SP - Beln Chen

23 Sple legan? SP - Beln Chen 3