CS434a/541a: Pattern Recognition Prof. Olga Veksler. Lecture 17

Transcription

1 CS434a/54a: Patter Recogto Prof. Olga Vesler Lecture 7

2 Today Paraetrc Usupervsed Learg Expectato Maxato (EM) oe of the ost useful statstcal ethods oldest verso 958 (Hartley) seal paper 977 (Depster et al.) ca also be used whe soe saples are ssg features

3 Usupervsed Learg I usupervsed learg, where we are oly gve saples x,, x wthout class labels Last lectures: oparaetrc approach (clusterg) Today, paraetrc approach assue paraetrc dstrbuto of data estate paraeters of ths dstrbuto uch harder tha the supervsed learg case

4 Paraetrc Usupervsed Learg Assue the data was geerated by a odel wth ow shape but uow paraeters P(x θ) Advatages of havg a odel Gves a eagful way to cluster data adust the paraeters of the odel to axe the probablty that the odel produced the observed data Ca sesbly easure f a clusterg s good copute the lelhood of data duced by clusterg Ca copare clusterg algorths whch oe gves the hgher lelhood of the observed data?

5 Paraetrc Supervsed Learg Let us recall supervsed paraetrc learg have classes have saples x,, x each of class,,, suppose D holds saples fro class probablty dstrbuto for class s p (x θ ) p (x θ ) p (x θ )

6 Paraetrc Supervsed Learg Use the ML ethod to estate paraeters θ fd θ whch axes the lelhood fucto F(θ ) p(d θ ) p( x θ ) x D F( θ ) or, equvaletly, fd θ whch axes the log lelhood l(θ ) l ( θ ) l p( D θ ) l p( x θ ) x D ˆ θ argax[ l p( D θ )] ˆ θ argax[ l p( D θ )] θ θ

7 Paraetrc Supervsed Learg ow the dstrbutos are fully specfed ca classfy uow saple usg MAP classfer p (x c )P(c )>p (x c )P(c ) p (x c )P(c )>p (x c )P(c ) p ( x ˆ ) θ p ( x ˆ ) θ

8 Paraetrc Usupervsed Learg I usupervsed learg, o oe tells us the true classes for saples. We stll ow have classes have saples x,, x each of uow class probablty dstrbuto for class s p (x θ ) Ca we detere the classes ad paraeters sultaeously?

9 Exaple: MRI Bra Segetato segetato Pcture fro M. Leveto I MRI bra age, dfferet bra tssues have dfferet testes Kow that bra has 6 aor types of tssues Each type of tssue ca be odeled by a Gaussa N(µ,σ ) reasoably well, paraeters µ,σ are uow Segetg (classfyg) the bra age to dfferet tssue classes s very useful do t ow whch age pxel correspods to whch tssue (class) do t ow paraeters for each N(µ,σ )

10 Mxture Desty Model Model data wth xture desty where P ( x c, ) P( c ) p( x θ ) p θ { θ } θ,..., copoet destes θ + ( c ) P( c ) P( c ) To geerate a saple fro dstrbuto p(x θ) frst select class wth probablty P(c ) the geerate x accordg to probablty law p(x c,θ ) xg paraeters p(x c,θ ) P(c ) P(c ) p(x c,θ ) P(c 3 ) p(x c 3,θ 3 )

11 Exaple: Gaussa Mxture Desty Mxture of 3 Gaussas p (x) p ( x ) p ( x ) N 0 0, 0 0 N [ 6,6 ], 4 p (x) p 3 ( x ) N [ 7, 7], 6 p 3 (x) ( x ) + 0.3p ( x ) 0.5p ( x ) p( x ) 0.p + 3

12 Mxture Desty ( x c, ) P( c ) p( x θ ) p θ P(c ),, P(c ) ca be ow or uow Suppose we ow how to estate θ,, θ ad P(c ),, P(c ) Ca brea apart xture p(x θ ) for classfcato To classfy saple x, use MAP estato, that s choose class whch axes P( c x, θ ) p( x c, θ )P( c ) probablty of copoet to geerate x probablty of copoet

13 ML Estato for Mxture Desty ( x c, ) P( c ) p( x θ, ρ) p θ p ( x c ), θ ρ Ca use Maxu Lelhood estato for a xture desty; eed to estate θ,, θ ρ P(c ),, ρ P(c ), ad ρ {ρ,, ρ } As the supervsed case, for the logarth lelhood fucto ( θ, ρ ) l p( D θ, ρ) l l p ( θ, ρ ) x l p ( x, θ ) c ρ

14 ML Estato for Mxture Desty l, c ρ ( θ ρ ) l p( x, θ ) eed to axe l(θ,ρ) wth respect to θ ad ρ As you ay have guessed, l(θ, ρ) s ot the easest fucto to axe If we tae partal dervatves wth respect to θ, ρ ad set the to 0, typcally we have a coupled olear syste of equato usually closed for soluto caot be foud We could use the gradet ascet ethod geeral, t s ot the greatest ethod to use, should oly be used as last resort There s a better algorth, called EM

15 Mxture Desty Before EM, let s loo at the xture desty aga p( x θ, ρ) p ( x c ), θ Suppose we ow how to estate θ,, θ ad ρ,,ρ Estatg the class of x s easy wth MAP, axe p( x c, θ ) P( c ) p( x c, θ ) ρ Suppose we ow the class of saples x,, x Ths s ust the supervsed learg case, so estatg θ,, θ ad ρ,,ρ s easy ˆ θ argax l [ p( θ )] D θ Ths s a exaple of chce-ad-egg proble ME algorth approaches ths proble by addg hdde varables ρ ˆ ρ D

16 Expectato Maxato Algorth EM s a algorth for ML paraeter estato whe the data has ssg values. It s used whe. data s coplete (has ssg values) soe features are ssg for soe saples due to data corrupto, partal survey respoces, etc. Ths scearo s very useful, covered secto 3.9. Suppose data X s coplete, but p(x θ) s hard to opte. Suppose further that troducg certa hdde varables U whose values are ssg, ad suppose t s easer to opte the coplete lelhood fucto p(x,u θ). The EM s useful. Ths scearo s useful for the xture desty estato, ad s subect of our lecture today Notce that after we troduce artfcal (hdde) varables U wth ssg values, case s copletely equvalet to case

17 EM: Hdde Varables for Mxture Desty p( x θ ) p( x c, θ ) ρ For splcty, assue copoet destes are p ( x c, θ ) exp σ π ( x µ ) σ assue for ow that the varace s ow eed to estate θ {µ,, µ } If we ew whch saple cae fro whch copoet (that s the class label), the ML paraeter estato s easy Thus to get a easer proble, troduce hdde varables whch dcate whch copoet each saple belogs to

18 EM: Hdde Varables for Mxture Desty For, ( ) 0, defe hdde varables () f saple was geerated by copoet otherwse { ( ) ( ) x, } x,..., () are dcator rado varables, they dcate whch Gaussa copoet geerated saple x Let { (),, () }, dcator r.v. correspodg to saple x Codtoed o, dstrbuto of x s Gaussa p where s s.t. () ( ) ( x, θ ~ N µ, σ )

19 EM: Jot Lelhood Let { (),, () }, ad {,, } The coplete lelhood s ( x,..., x,,..., ) p X, θ ) p θ ( p ( x, θ ) p( ) gaussa part of ρ c p (, θ ) x If we actually observed, the log lelhood l[p(x, θ)] would be trval to axe wth respect to θ ad ρ The proble, s, of course, s that the values of are ssg, sce we ade t up (that s s hdde)

20 EM Dervato Istead of axg l[p(x, θ)] the dea behd EM s to axe soe fucto of l[p(x, θ)], usually ts expected value E [ lp( X, θ )] If θ aes l[p(x, θ)] large, the θ teds to ae E[l p(x, θ)] large the expectato s wth respect to the ssg data that s wth respect to desty p( X,θ) however θ s our ultate goal, we do t ow θ!

21 EM Algorth EM soluto s to terate. start wth tal paraeters θ (0) terate the followg step utl covergece E. copute the expectato of log lelhood wth respect to curret estate θ (t) ad X. Let s call t Q(θ θ (t) ) Q [ ] ( ( t )) ( t ) θ θ E lp( X, θ ) X, θ M. axe Q(θ θ (t) ) ( θ θ ) ( t ) ( t ) θ + argaxq θ

22 EM Algorth: Pcture lp ( X θ ) optal value for θ we d le to fd t but optg p(x θ) s very dffcult θ

23 EM Algorth: Pcture lp( X, θ ) uobserved correspodg to observed data X Ths curve correspods to the correct, we should opte for but s ot observed θ for xture estato, there are curves, each curve correspods to a partcular assget of saples to classes

24 EM Algorth: Pcture lp( X, θ ) E ( ) [ lp( X, θ ) X, θ ] t θ E (θ)

25 EM Algorth It ca be prove that EM algorth coverges to the local axu of the log-lelhood lp ( X θ ) Why s t better tha gradet ascet? Covergece of EM s usually sgfcatly faster, the begg, very large steps are ade (that s lelhood fucto creases rapdly), as opposed to gradet ascet whch usually taes ty steps gradet descet s ot guarateed to coverge recall all the dffcultes of choosg the approprate learg rate

26 EM for Mxture of Gaussas: E step Let s coe bac to our exaple p( x θ ) p( x c, θ ) ρ p ( x c, θ ) exp σ π ( x µ ) eed to estate θ {µ,, µ } ad ρ,, ρ ( ) 0 σ for,, defe () f saple was geerated by copoet otherwse as before, { (),, () }, ad {,, } We eed log-lelhood of observed X ad hdde l p( X, θ ) l p(, θ ) l p( x, θ ) P( ) x

27 EM for Mxture of Gaussas: E step We eed log-lelhood of observed X ad hdde ( ) ( ) x p X p, l, l θ θ ( ) ( ) P x p, l θ Frst let s rewrte ( ) ( ) P x p θ, ( ) ( ) P x p θ, ( ) ( ) ( ) ( ) [ ] ( ) P x p, θ ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) f P, x p f P, x p θ θ ( ) ( ) ( ) ( ) P x exp σ µ π σ

28 EM for Mxture of Gaussas: E step log-lelhood of observed X ad hdde s l p ( X, θ ) l p( x, θ ) P( ) l exp σ π ( x µ ) ( ) σ P ( ) ( ) l exp σ π ( x µ ) ( ) σ P ( ) ( ) l σ π µ σ ( ) ( ) ( ) ( ) P x + lp ( saple x fro class ) P( c ) ρ ( ) ( x µ ) l σ π σ + l ρ

29 EM for Mxture of Gaussas: E step log-lelhood of observed X ad hdde s l p ( ) ( ) ( x µ ) X, θ l σ π σ + l ρ For the E step, we ust copute ( t ) t t t t ( ) Q θ θ Q θ µ,..., µ, ρ,..., ρ E [ lp( X, θ ) X, θ ] t ( ) ( ) ( ) ( ) ( ) ( ) E l σ π ( ) ( ) ( t ) x µ σ + l ρ E X a x + b ae X + [ x ] b E + l ρ σ π σ ( ) [ ] ( x µ ) l

30 EM for Mxture of Gaussas: E step Q + l ρ σ π σ ( t ) ( ) ( θ θ ) E [ ] ( x µ ) l eed to copute E [ () ] the above expresso E [ ( )] ( ) ( t ) * P 0 ( ) ( ( ) ( t ) θ, x + * P, x ) 0 θ ( ) ( t ) ( θ x ) P, p ( t ) ( ) ( ) ( t ) ( x θ, ) P θ ( t ) p ( ) ( x θ ) P ρ ( x µ ) ( t ) ( t ) ( t ) ( ) ( ) ( t ) ( x θ, ) P θ exp ( ) ρ exp σ ρ exp σ ( x µ ) ( t ) ( t ) ( x µ ) ( t ) ( t ) we are fally doe wth the E step for pleetato, ust eed to copute E [ () ] s do t eed to copute Q

31 EM for Mxture of Gaussas: M step Q ( t ) ( ) ( θ θ ) E [ ] ( x µ ) l + l ρ σ π σ Need to axe Q wth respect to all paraeters Frst dfferetate wth respect to µ µ Q ( t ) ( θ θ ) E ew µ ( ) [ ] ( ) x µ µ σ 0 [ ] ( t + ) ( ) E x the ea for class s weghted average of all saples, ad ths weght s proportoal to the curret estate of probablty that the saple belogs to class

32 EM for Mxture of Gaussas: M step Q ( t ) ( ) ( θ θ ) E [ ] ( x µ ) l + l ρ σ π σ For ρ we have to use Lagrage ultplers to preserve costrat ρ ( t ) Thus we eed to dfferetate F( λ, ρ ) Q θ θ ( ) λ ρ ρ F ρ ( ) ( λ, ρ ) E [ ] λ 0 E ( ) [ ] λρ 0 Sug up over all copoets: Sce E ( ) [ ] ρ ( ) [ ] E ad ρ we get ( t + ) ( ) E [ ] λ λρ

33 EM Algorth The algorth o ths slde apples ONLY to uvarate gaussa case wth ow varaces. Radoly tale µ,, µ, ρ,, ρ (wth costrat Σρ ) terate utl o chage µ,, µ, ρ,, ρ E. for all,, copute E ( ) [ ] ρ exp σ ρ exp σ ( x µ ) ( x µ ) M. for all, do paraeter update µ E [ ( )] x ρ E [ ( )]

34 EM Algorth For the ore geeral case of ultvarate Gaussas wth uow eas ad varaces ( ) [ ] ( ) ( ), x p, x p E Σ µ ρ Σ µ ρ E step: ( ) ( ) ( ) t / / d x x exp ) (, x p µ Σ µ Σ π Σ µ where ( ) [ ] ( ) [ ] E x E µ ( ) [ ]( )( ) ( ) [ ] Σ T E x x E µ µ ( ) [ ] E ρ M step:

35 EM Algorth ad K-eas -eas ca be derved fro EM algorth Settg xg paraeters equal for all classes, E ( ) [ ] ρ exp σ ρ exp σ ( x µ ) ( x µ ) exp σ exp σ ( x µ ) ( x µ ) If we let σ E ( ) [ ], the f, x µ > x µ 0 otherwse so at the E step, for each curret ea, we fd all pots closest to t ad for ew clusters at the M step, we copute the ew eas sde curret clusters µ E x [ ( )]

36 EM Gaussa Mxture Exaple

37 EM Gaussa Mxture Exaple After frst terato

38 EM Gaussa Mxture Exaple After secod terato

39 EM Gaussa Mxture Exaple After thrd terato

40 EM Gaussa Mxture Exaple After 0th terato

41 EM Exaple Exaple fro R. Guterre-Osua Trag set of 900 exaples forg a aulus Mxture odel wth 30 Gaussa copoets of uow ea ad varace s used Trag: Italato: eas to 30 rado exaples covarace atrces taled to be dagoal, wth large varaces o the dagoal (copared to the trag data varace) Durg EM trag, copoets wth sall xg coeffcets were tred Ths s a trc to get a ore copact odel, wth fewer tha 30 Gaussa copoets

42 EM Exaple fro R. Guterre-Osua

43 EM Texture Segetato Exaple Fgure fro Color ad Texture Based Iage Segetato Usg EM ad Its Applcato to Cotet Based Iage Retreval,S.J. Beloge et al., ICCV 998

44 EM Moto Segetato Exaple Three fraes fro the MPEG flower garde sequece Fgure fro Represetg Iages wth layers,, by J. Wag ad E.H. Adelso, IEEE Trasactos o Iage Processg, 994, c 994, IEEE

45 Suary Advatages If the assued data dstrbuto s correct, the algorth wors well Dsadvatages If assued data dstrbuto s wrog, results ca be qute bad. I partcular, bad results f use correct uber of classes (.e. the uber of xture copoets)