10/7/14. Mixture Models. Comp 135 Introduction to Machine Learning and Data Mining. Maximum likelihood estimation. Mixture of Normals in 1D

Transcription

1 Comp 35 Introducton to Machn Larnng and Data Mnng Fall 204 rofssor: Ron Khardon Mxtur Modls Motvatd by soft k-mans w dvlopd a gnratv modl for clustrng. Assum thr ar k clustrs Clustrs ar not rqurd to hav th sam numbr of ponts And not rqurd to hav th sam shap Computr Scnc Tufts Unvrsty Mxtur of Normals n D Rpat for =,...,N ck clustr Id z from dscrt dstrbuton wth paramtrs p,p 2,...,p k Not: z 2 {, 2,...,k} ck th xampl x from normal dstrbuton wth paramtrs µ z Exampl: whn z =3usngµ 3 and 3 Gvn a datast gnratd by ths procss th clustrng task s to dntfy th paramtrs {p,µ, } =,...,k z Maxmum lklhood stmaton Frst analyz assumng z ar known Convnnt notaton: rprsnt th numbr z as a unt vctor bt squnc Exampl: k=4 z =) 000 z =2) 000 z =3) 000 z =4) 000 Notaton: z, s th bt wthn z z =2) 000 ) z,2 = z,3 =0 Maxmum lklhood stmaton Maxmum lklhood stmaton Frst analyz assumng z ar known Th Complt Data ncluds all th x,z Th Lklhood Data =(x,z ), (x 2,z 2 ),...,(x N,z N ) L = Y p(z )p(x z,µ z ) = Y (/k) p 2 2 (x µz )2 = Y (/k) p 2 2 z,(x µ)2 Notaton trck: xactly on trm rmans from th sum!

2 Maxmum lklhood stmaton L = Y (/k) p LogL = µ =...=0 ) µ = z, x z, 2 2 z,(x µ)2 2 2 z, (x µ ) 2 W solvd ths n th last lctur Maxmum lklhood stmaton Frst analyz assumng z ar known Th Complt Data ncluds all th x,z Data =(x,z ), (x 2,z 2 ),...,(x N,z N ) Th Obsrvd Data ncluds all th Data = x,x 2...,x N Th Eq for th lklhood nds to sum out (margnalz) ovr th z. No smpl closd form. x Th EM Algorthm Th EM Algorthm A gnral algorthm for maxmzng margnal lklhood.. maxmzng lklhood whn w hav hddn random varabls Th algorthm has a smpl form whn appld to mxtur modls (and som othr modls) W frst xplan ths spcal form and thn show how to drv t from th gnral schm of th EM algorthm EM s an tratv algorthm Intalz probablty modl p us p to calculat an mprovd modl p St p =p Untl no furthr mprovmnt EM Algorthm for Mxtur Modls [E] Calculat usng p f, = E p(z,{µ 0`})[z, ]=p(z = {µ 0`},Data) [M] Estmat p paramtrs usng max lklhood rplacng th unknown z, by f, EM for Mxturs n D [E] Calculat f, = E p(z,{µ 0`})[z, ]=p(z = {µ 0`},Data) f, = p((z = ) and x ) p(x ) = p((z = ) and x ) ` p((z = `) and x ) = (/k) p `(/k) p 2 2 (x µ0 )2 2 2 (x µ0`)2 Frst part holds for any mxtur modl. 2

3 EM for Mxturs n D [M] Estmat paramtrs usng max lklhood rplacng th unknown z, by µ = z, x z ), µ 00 = f, x f, f, Gnral form of EM Dfn an auxlary functon Q(p,p ) Rlatv to obsrvd varabls O and hddn varabls H Q(p 0,p 00 )=E p0 (H O)[log p 00 (H, O)] Th EM Algorthm Th EM Algorthm EM s an tratv algorthm Intalz probablty modl p ck p so as to maxmz Q(p,p ) St p =p Untl no furthr mprovmnt Fact: If Q(p,p ) > Q(p,p ) [ths mans that p ncrass th valu of Q()] Thn p (obsrvd) > p (obsrvd) Thrfor tratons of EM ncras th lklhood and th algorthm convrgs to a (possbly local) maxmum of th lklhood. Rmndr: Maxmum lklhood EM for Mxturs n D L = Y (/k) p LogL = µ =...=0 ) µ = z, x z, 2 2 z,(x µ)2 2 2 z, (x µ ) 2 W solvd ths n th last lctur Dvlop algorthm from th gnral tmplat Q(p 0,p 00 )=E p0 (H O)[log p 00 (H, O)] Q(p 0,p 00 ) = E p0 (Z )[log p 00 (Z, )] = E p 0 (Z )[const 2 2 z, (x µ 00 ) 2 ] What s th random varabl n ths xpctaton? Only z, 3

4 Q(p 0,p 00 ) EM for Mxturs n D = E p0 (Z )[log p 00 (Z, )] = E p 0 (Z )[const 2 2 z, (x µ 00 ) 2 ] =const =const E p0 (Z )[z, ](x µ 00 ) 2 f, (x µ 00 ) 2 W gt xactly th sam Eq as bfor! EM Algorthm for Mxtur Modls [E] Calculat usng p f, = E p(z,{µ 0`})[z, ]=p(z = {µ 0`},Data) [M] Estmat p paramtrs usng max lklhood rplacng th unknown z, by Usng th sam mthodology on any mxtur modl (not ust Gaussan) ylds th sam tmplat. f, Sm-Suprvsd Naïv Bays Modl Naïv Bays: robablstc modl wth strong smplfyng assumptons Illustratng applcaton: txt catgorzaton whr w hav data for (documnt,labl ) What f w hav many documnts but labls for only a fw of thm? Can th unlabld documnts hlp? Sm-Suprvsd Naïv Bays Modl What f w hav many documnts but labls for only a fw of thm? Can th unlabld documnts hlp? Bfor xplorng ths quston w wll dvlop th EM algorthm for ths modl whr th labls ar not known Rcall: Naïv Bays Modl Rcall: Naïv Bays Modl Each class nducs a dstrbuton ovr faturs. Faturs ar condtonally ndpndnt gvn th class On slds I us modl for bnary faturs p(z = ) =p p(x,` = class ) =q,` p(x class ) = Ỳ q x,`,` ( q ( x,`),`) p(z = and x )=p Y p(z and x )= Y " ` p Y ` q x,`,` ( q ( x,`),`) q x,`,` ( q ( x,`),`) # z, 4

5 Rcall: Maxmum Lklhood p = p(z = ) = numbr of xampls wth class numbr of xampls q,` = p(x,` = z = ) = num of x wth class and x,` = numbr of xampls wth class Naïv Bays for Txt Classfcaton Classs: Ys, No Faturs: word-slot n documnt has valu word- n lxcon Faturs hav a hug numbr of valus Numbr of faturs s documnt lngth In th applcaton th faturs ar not bnary. In th applcaton w assumd that all faturs (word slots) hav th sam dstrbuton. q, = q,2 =...= q,m W kp to th bnary modl n th slds. Naïv Bays as Mxtur Modl EM Algorthm Rpat for =,...,N ck clustr Id z from dscrt dstrbuton wth paramtrs p,p 2,...,p k ck th xampl x from Nav Bays dstrbuton wth paramtrs q z Complt Data Lklhood L = Y " Y Y p q x,`,` ( ` q (,`) x,`) Log Lklhood # z, LogL = z, (log p + ` x,` log q,` +( x,`) log( q,`)) EM Algorthm EM Algorthm Complt Data Lklhood L = Y " Y Y p q x,`,` ( q ( x,`),`) ` Log Lklhood usng p notaton LogL = z, (log p 00 + ` # z, x,` log q 00,` +( x,`) log( q 00,`)) Complt Data Lklhood L = Y " Y Y p q x,`,` ( q ( x,`),`) ` Log Lklhood usng p notaton LogL = Q(p,p ) z, (log p 00 + ` Q(p 0,p 00 )=E p0 (Z )[log p 00 (Z, )] # z, x,` log q 00,` +( x,`) log( q 00,`)) What s th random varabl n ths xpctaton? Only z, 5

6 EM Algorthm Sam stps as bfor Calculat f, Solv Maxmum lklhood usng f, nstad of z, EM Algorthm E Stp: Calculatng f, f, = E p 0 (Z )[z, ]= p0 (z = and x ) c p0 (z = c and x ) = p0 Q` q0x,`,` ( q0 ( x,`),`) c p0 c Q` q0x,` c,` ( q0 ( x,`) c,`) EM Algorthm EM Algorthm for Naïv Bays M Stp: Maxmzng wrt to p and q,ll Q = f, (log p 00 + ` x,` log q,` 00 +( x,`) log( q,`)) 00 p 00 = q 00,` = [skppng dtals of drvatvs] f, N f,x,` f, Calculat: Calculat: f, = p 00 = f, N q 00,` = p0 Q` q0x,` c p0 c f,x,` f, Assgn: p ß p and q ß q,` ( q0 ( x,`),`) Q` q0x,` c,` ( q0 ( x,`) c,`) Sm-Suprvsd Naïv Bays Modl What f w hav many documnts but labls for only a fw of thm? Can th unlabld documnts hlp? 00% 90% 80% 70% 20 nwgroups data 0000 unlabld documnts No unlabld documnts Us EM: for xampls whr z s known us f, =z, nstad of stmatng t Nothng ls changs n th algorthm! Accuracy 60% 50% 40% 30% 20% 0% 0% Numbr of Labld Documnts [From Ngam t all MLJ 999.] 6

7 20 nwgroups data 00% 90% 80% 70% 3000 labld documnts 600 labld documnts 300 labld documnts 40 labld documnts 40 labld documnts Accuracy 60% 50% 40% 30% 20% 0% 0% Numbr of Unlabld Documnts [From Ngam t all MLJ 999.] 7