Mixture models (cont d)

Transcription

1 6.867 Machie learig, lecture 5 (Jaakkola) Lecture topics: Differet types of ixture odels (cot d) Estiatig ixtures: the EM algorith Mixture odels (cot d) Basic ixture odel Mixture odels try to capture ad resolve observable abiguities i the data. E.g., a copoet Gaussia ixture odel P (x; θ) = P (j)n(x; µ j, Σ j ) () j= The paraeters θ iclude the ixig proportios (prior distributio) {P (j)}, eas of copoet Gaussias {µ j }, ad covariaces {Σ j }. The otatio {P (j)} is a shorthad for {P (j), j =,..., }. To geerate a saple x fro such a odel we would first saple j fro the prior distributio {P (j)}, the saple x fro the selected copoet N(x; µ j, Σ j ). If we geerated saples, the we would get potetially overlappig clusters of poits, where each cluster ceter would correspod to oe of the eas µ j ad the uber of poits i the clusters would be approxiately P (j). This is the type of structure i the data that the ixture odel is tryig to capture if estiated o the basis of observed x saples. Studet exa odel: -year We ca odel vectors of exa scores with ixture odels. Each x is a vector of scores fro a particular studet ad saples correspod to studets. We expect that the populatio of studets i a particular year cosists of differet types (e.g., due to differeces i backgroud). If we expect each type to be preset with a overall probability P (j), the each studet score is odeled as a ixture P (x θ) = P (x θ j )P (j) () j= where we su over the types (weighted by P (j)) sice we do t kow what type of studet t is prior to seeig their exa score x t. Cite as: Toi Jaakkola, course aterials for Machie Learig, Fall 6. MIT OpeCourseWare ( Massachusetts Istitute of Techology. Dowloaded o [DD Moth YYYY].

2 6.867 Machie learig, lecture 5 (Jaakkola) If there are studets takig the exa a particular year, the the likelihood of all the studet score vectors, D = {x,..., x }, would be ( ) L(D ; θ) = P (x t θ) = P (x θ j )P (j) (3) Studet exa odel: K-years t= t= j= Suppose ow that we have studet data fro K years of offerig the course. I year k we have k studets who took the course. Let x k,t deote the score vector for a studet t i year k. Note that t is just a idex to idetify saples each year ad the sae idex does ot iply that the sae studet took the course ultiple years. We ca ow assue that the uber of studet types as well as P (x θ j ) reai the sae fro year to year (the paraeters θ j are the sae for all years). However, the populatio of studets ay easily chage fro year to year, ad thus the prior probabilities over the types have to be set differetly. Let P (j k) deote the prior probabilities over the types i year k (all of these would have to be estiated of course). Now, accordig to our ixture distributio, we expect exaple scores for studets i year k be sapled fro P (x k, θ) = P (x θ j )P (j k) (4) j= The likelihood of all the data, across K years, D = {D,..., D K }, is give by ( ) K k K k L(D; θ) = P (x k,t k, θ) = P (x k,t θ j )P (j k) (5) k= t= k= t= j= The paraeters θ here iclude the ixig portios {P (j k)} that chage fro year to year i additio to {θ j }. Collaborative filterig Mixture odels are useful also i recoeder systes. Suppose we have users ad ovies ad our task is to recoed ovies for users. The users have each rated a sall fractio of ovies ad our task is to fill-i the ratig atrix, i.e., provide a predicted ratig for each user across all ovies. Such a predictio task is kow as a collaborative filterig proble (see Figure ). Say the possible ratigs are r ij {,..., 5} (i.e., how ay stars you assig to each ovie). We will use i to idex users ad j for ovies; a ratig r ij, if provided, specifies how user Cite as: Toi Jaakkola, course aterials for Machie Learig, Fall 6. MIT OpeCourseWare ( Massachusetts Istitute of Techology. Dowloaded o [DD Moth YYYY].

3 6.867 Machie learig, lecture 5 (Jaakkola) 3 ovies j 3 r ij users i 5 5 Figure : Partially observed ratig atrix for a collaborative filterig task. i rated ovie j. Sice oly a sall fractio of ovies are rated by each user, we eed a way to idex these eleets of the user/ovie atrix: we say (i, j) I D if ratig r ij is available (observed). D deotes all the observed ratigs. We ca build o the previous discussio o ixture odels. We ca represet each ovie as a distributio over ovie types z {,..., K }. Siilarly, a user is represeted by a distributio over user types z u {,..., K u }. We do ot assue that each ovie correspods to a sigle ovie type across all users. Istead, we iterpret the distributio over ovie types as a bag of features correspodig to the ovie ad we resaple fro this bag i the cotext of each user. This is aalogous to predictig exa scores for studets i a particular year (we did t assue that all the studets had the sae type). We also ake the sae assuptio about user types, i.e., that the type is sapled fro the bag for each ratig, resultig potetially i differet types for differet ovies. Sice all the uobserved quatities are sued over, we do ot explicitly assig ay fixed ovie/user type to a ratig. We iagie geeratig the ratig r ij associated with (i, j) eleet of the ratig atrix as follows. Saple a ovie type fro P (z j), saple a user type fro P (z u i), the saple a ratig r ij with probability P (r ij z u, z ). All the probabilities ivolved have to estiated fro the available data. Note that we will resaple ovie ad user types for each ratig. The resultig ixture odel for ratig r ij is give by K u K P (r ij i, j, θ) = P (r ij z u, z )P (z u i)p (z j) (6) z u= z = where the paraeters θ refer to the appig fro types to ratigs {P (r z u, z )} ad the user ad ovie specific distributios {P (z u i)} ad {P (z j)}, respectively. The likelihood Cite as: Toi Jaakkola, course aterials for Machie Learig, Fall 6. MIT OpeCourseWare ( Massachusetts Istitute of Techology. Dowloaded o [DD Moth YYYY].

4 6.867 Machie learig, lecture 5 (Jaakkola) 4 of the observed data D is L(D; θ) = P (r ij i, j, θ) (7) (i,j) I D Users rate ovies differetly. For exaple, soe users ay oly use a part of the scale (e.g., 3, 4 or 5) while others ay be bi-odal, ratig ovies either very bad or very good 5. We ca iprove the odel by assuig that each user has a ratig style s {,..., K s }. The styles are assued to be the sae for all users, we just do t kow how to assig each user to a particular style. The prior probability that ay radoly chose user would have style s is specified by P (s). These paraeters are coo to all users. We also assue that the ratig predictios P (r ij z u, z ) associated with user/ovie types ow deped o the style s as well: P (r ij z u, z, s). We have to be a bit careful i writig the likelihood of the data. All the ratigs of oe user have to coe fro oe ratig style s but we ca su over the possibilities. As a result, the likelihood of the observed ratigs is odified to be likelihood of user i s ratigs with style s [ { ( }} ){ ] K s K u K L (D; θ) = P (s) P (r ij z u, z, s)p (z u i)p (z j) (8) i= s= j:(i,j) I D z u= z = The odel does ot actually ivolve that ay paraeters to estiate. There are exactly {P (s)} {P (r z u,z,s)} {P (z u i)} {P (z j)} {}}{{}}{{}}{{}}{ (K s ) + (5 )K u K K s + (K u ) + (K ) (9) idepedet paraeters i the odel. A realistic odel would iclude, i additio, a predictio of the issig eleets i the ratig atrix, i.e., a odel of why the etry was issig (a user failed to rate a ovie they had see, ot see but could, chose ot to see, etc.). Estiatig ixtures: the EM-algorith We have see a uber of differet types of ixture odels. The advatage of ixture odels lies i their ability to icorporate ad resolve abiguities i the data, especially Cite as: Toi Jaakkola, course aterials for Machie Learig, Fall 6. MIT OpeCourseWare ( Massachusetts Istitute of Techology. Dowloaded o [DD Moth YYYY].

5 6.867 Machie learig, lecture 5 (Jaakkola) 5 i ters of uidetified sub-groups. However, we ca ake use of the oly if we ca estiate such odels easily fro the available data. Coplete data. The siplest way to uderstad how to estiate ixture odels is to start by pretedig that we kew all the sub-typig (copoet) assigets for each available data poit. This is aalogous to kowig the label for each exaple i a classificatio cotext. We do t actually kow these (they are uobserved i the data) but solvig the estiatio proble i this cotext will help us later o. Let s begi with the siple Gaussia ixture odel i Eq.(), P (x; θ) = P (j)n(x; µ j, Σ j ) () j= ad preted that each observatio x,..., x also had iforatio about the copoet that was resposible for geeratig it, i.e., we also observed j,..., j. This additioal copoet iforatio is coveiet to iclude i the for of - assigets δ(j t) where δ(j t t) = ad δ(j t) = for all j j t. The log-likelihood of this coplete data is [ ] l(x,..., x, j,..., j ; θ) = log P (j t )N(x t ; µ jt, Σ jt ) () t= [ ] = δ(j t) log P (j)n(x t ; µ j, Σ j ) () t= j= ( ) = δ(j t) log P (j) j= t= ( ) + δ(j t) log N(x t ; µ j, Σ j ) (3) j= t= It s iportat to ote that i tryig to axiize this log-likelihood, all the Gaussias ca be estiated separately fro each other. Put aother way, because our preted observatios are coplete, we ca estiate each copoet fro oly data pertaiig to it; the proble of resolvig which copoet should be resposible for which data poits is ot Cite as: Toi Jaakkola, course aterials for Machie Learig, Fall 6. MIT OpeCourseWare ( Massachusetts Istitute of Techology. Dowloaded o [DD Moth YYYY].

6 6.867 Machie learig, lecture 5 (Jaakkola) 6 preset. As a result, the axiizig solutio ca be writte i closed for: ˆP (j) = ˆµ j = ˆΣ j = ˆ(j), where ˆ(j) = δ(j t) t= (4) δ(j t)x t ˆ(j) t=] (5) ˆ(j) δ(j t)(x t ˆµ j )(x t ˆµ j ) T (6) t=] I other words, the prior probabilities siply recover the epirical fractios fro the observed j,..., j, ad each Gaussia copoet is estiated i the usual way (evaluatig the epirical ea ad the covariace) based o data poits explicitly assiged to that copoet. So, the estiatio of ixture odels would be very easy if we kew the assigets j,..., j. Icoplete data. What chages if we do t kow the assigets? We ca always guess what the assigets should be based o the curret settig of the paraeters. Let θ (l) deote the curret (iitial) paraeters of the ixture distributio. Usig these paraeters, we ca evaluate for each data poit x t the posterior probability that it was geerated fro copoet j: P (l) (l) (j)n(x t ; µ j, Σ (l) j ) P (j x t, θ (l) ) = = P (l) (j )N(x t ; µ (l) (l) j = j, Σ j ) P (l) (l) (j)n(x t ; µ j, Σ (l) j ) P (x t ; θ (l) ) Istead of usig the - assigets δ(j t) of data poits to copoets we ca use soft assigets p (l) (j t) = P (j x t, θ (l) ). By substitutig these i the above closed for estiatig equatios we get a iterative algorith for estiatig Gaussia ixtures. The algorith is iterative sice the soft posterior assigets were evaluated based o the curret settig of the paraeters θ (l) ad ay have to revised later o (oce we have a better idea of where the clusters are i the data). The resultig algorith is kow as the Expectatio Maxiizatio algorith (EM for short) ad applies to all ixture odels ad beyod. For Gaussia ixtures, the EM-algorith ca be writte as (Step ) Iitialize the Gaussia ixture, i.e., specify θ (). A siple iitializatio () cosists of settig P () (j) = /, equatig each µ j with a radoly chose data () poit, ad akig all Σ j equal to the overall data covariace. (7) Cite as: Toi Jaakkola, course aterials for Machie Learig, Fall 6. MIT OpeCourseWare ( Massachusetts Istitute of Techology. Dowloaded o [DD Moth YYYY].

7 6.867 Machie learig, lecture 5 (Jaakkola) 7 (E-step) Evaluate the posterior assiget probabilities p (l) (j t) = P (j x t, θ (l) ) based o the curret settig of the paraeters θ (l). (M-step) Update the paraeters accordig to P (l+) (j) = (l+) µ j = Σ (l+) = ˆ( j), where ˆ(j) = p (l) (j t) (8) ˆ( j) t= p (l) (j t)x t (9) t= p (l) (j t)(x t µ (l+) )(x t µ (l+) ) T () ˆ(j) j j j t= Perhaps surprisigly, this iterative algorith is guarateed to coverge ad each iteratio icreases the log-likelihood of the data. I other words, if we write the l(d; θ (l) ) = P (x t ; θ (l) ) () t= l(d; θ () ) < l(d; θ () ) < l(d; θ () ) <... () util covergece. The ai dowside of this algorith is that we are oly guarateed to coverge to a locally optial solutio where d/dθ l(d; θ) =. I other words, there could be a settig of the paraeters for the ixture distributio that leads to a higher log-likelihood of the data. For this reaso, the algorith is typically ru ultiple ties (recall the rado iitializatio of the eas) so as to esure we fid a reasoably good solutio, albeit perhaps oly locally optial. Exaple. Cosider a siple ixture of two Gaussias. Figure deostrates how the EM-algorith chages the Gaussia copoets after each iteratio. The ellipsoids specify oe stadard deviatio distaces fro the Gaussia ea. The ixig proportios P (j) are ot visible i the figure. Note that it takes ay iteratios for the algorith to resolve how to properly assig the data poits to ixture copoets. At covergece, the assigets are still soft (ot -) but evertheless clearly divide the resposibilities of the two Gaussia copoets across the clusters i the data. I fact, the highest likelihood for Gaussia ixtures is always. This happes whe oe of the Gaussias cocetrates aroud a sigle data poit. We do ot look for such solutios, however, ad they ca be reoved by costraiig the covariace atrices or via regularizatio. The real copariso is to a o-trivial ixture that achieves the highest log-likelihood. Cite as: Toi Jaakkola, course aterials for Machie Learig, Fall 6. MIT OpeCourseWare ( Massachusetts Istitute of Techology. Dowloaded o [DD Moth YYYY].

8 6.867 Machie learig, lecture 5 (Jaakkola) iitializatio iteratio iteratio iteratio 4 iteratio 6 iteratio Figure : A exaple of the EM algorith with a two-copoet ixture of Gaussias odel. Cite as: Toi Jaakkola, course aterials for Machie Learig, Fall 6. MIT OpeCourseWare ( Massachusetts Istitute of Techology. Dowloaded o [DD Moth YYYY].