Data Mining Techniques

Transcription

1 Data Mining Techniques CS Sectin 2 - Spring 2017 Lecture 7 Jan-Willem van de Meent (credit: David Blei)

2 Review: K-means Clustering μ1 Objective: Sum f Squares μ2 µ k One-ht assignment Center fr cluster k μ3 Alternate between tw steps 1. Minimize SSE w.r.t. zn 2. Minimize SSE w.r.t. μk

3 Review: Prbabilistic K-means Generative Mdel z n Discrete( ) x n z n = k Nrm(µ k, k ) Questins 1. What is lg p(x, z μ, Σ, π)? 2. Fr what chice f π and Σ d we recver K-means? Same as K-means when: k = 1/K k = 2 I

4 Review: Prbabilistic K-means Assignment Update Parameter Updates Idea: Replace hard assignments with sft assignments N k := P N n=1 z nk N = [ = ] = =(N 1 /N,...,N K /N) 1 µ k = P N z N k n=1 x nk n P 1 P N P k = 1 N k P N n=1 z nk (x n µ k )(x n µ k ) >

5 Review: Sft K-means Sft Assignment Update Parameter Updates Idea: Replace hard assignments with sft assignments N k := P N n=1 nk = PN [ N,...,N = ] N = P =(N 1 /N,...,N K /N)) 1 P µ k = N z x N k n=1 nk n P = k = 1 N N k n=1 nk (x n µ k )(x n µ k ) >

6 Review: Lwer Bund n Lg Likelihd (multiplicatin by 1)

7 Review: Lwer Bund n Lg Likelihd (multiplicatin by 1) (multiplicatin by 1)

8 Review: Lwer Bund n Lg Likelihd (multiplicatin by 1) (multiplicatin by 1) (Bayes rule)

9 Review: Lwer Bund n Lg Likelihd (multiplicatin by 1) (multiplicatin by 1) (Bayes rule)

10 Review: Lwer Bund n Lg Likelihd

11 Review: Lwer Bund n Lg Likelihd

12 Review: EM fr Gaussian Mixtures Generative Mdel z n Discrete( ) x n z n = k Nrm(µ k, k ) Expectatin Maximizatin Initialize θ Repeat until cnvergence 1. Expectatin Step 2. Maximizatin Step

13 TOPIC MODELS Brrwing frm: David Blei (Clumbia)

14 Wrd Mixtures Generative mdel f Latent Dirichlet allcatin (LDA) Idea: Mdel text as a mixture ver wrds (ignre rder) Tpics gene dna genetic.,, life 0.02 evlve 0.01 rganism 0.01.,, brain neurn nerve data 0.02 number 0.02 cmputer 0.01.,, Each tpic is a distrib Wrds: Tpics: Simple intuitin: Dcuments exhibit multiple tpics. Each dcument is a

15 EM fr Wrd Mixtures Generative Mdel Expectatin Maximizatin Initialize θ Repeat until cnvergence 1. Expectatin Step 2. Maximizatin Step

16 EM fr Wrd Mixtures Generative Mdel E-step: Update assignments M-step: Update parameters

17 Tpic Mdeling Tpics gene 0.04 dna 0.02 genetic 0.01.,, Dcuments Tpic prprtins and assignments life 0.02 evlve 0.01 rganism 0.01.,, brain 0.04 neurn 0.02 nerve data 0.02 number 0.02 cmputer 0.01.,, Each tpic is a distributin ver wrds Each dcument is a mixture ver tpics Each wrd is drawn frm ne tpic distributin

18 Tpic Mdeling Tpics gene 0.04 dna 0.02 genetic 0.01.,, Dcuments Tpic prprtins and assignments life 0.02 evlve 0.01 rganism 0.01.,, brain 0.04 neurn 0.02 nerve data 0.02 number 0.02 cmputer 0.01.,, Wrds: Tpics:

19 EM fr Tpic Mdels (PLSI/PLSA*) Generative Mdel E-step: Update assignments M-step: Update parameters *(Prbabilistic Latent Semantic Indexing, a.k.a. Prbabilistic Latent Semantic Analysis)

20 Tpic Mdels with Prirs Generative Mdel (with prirs) Maximum a Psteriri E-step: Update assignments M-step: Update parameters

21 Latent Dirichlet Allcatin (a.k.a. PLSI/PLSA with prirs) Prprtins parameter Per-wrd tpic assignment Per-dcument tpic prprtins Observed wrd Tpics Tpic parameter d Z d,n W d,n N k D K η

22 Intermezz: Dirichlet Distributin

23 Intermezz: Dirichlet Distributin

24 Intermezz: Cnjugacy Likelihd (discrete) Prir (Dirichlet) Questin: What distributin is the psterir? Mre examples:

25 MAP estimatin fr LDA Generative Mdel (with prirs) Maximum a Psteriri E-step: Update assignments M-step: Update parameters

26 Variatinal Inference Idea: Maximize Evidence Lwer Bund (ELBO) Maximizing the ELBO is equivalent t minimizing the KL divergence

27 Variatinal EM Use Factrized Apprximatin fr q(z,β,θ) Discrete Dirichlet Dirichlet Variatinal E-step: Maximize w.r.t. φ (expectatins clsed frm fr Dirichlet distributins) Variatinal M-step: Maximize w.r.t. λ and γ (analgus t MAP estimatin)

28 Variatinal EM Use Factrized Apprximatin fr q(z,β,θ) Discrete Dirichlet Dirichlet Variatinal E-step: Maximize w.r.t. φ (expectatins clsed frm fr Dirichlet distributins) Variatinal M-step: Maximize w.r.t. λ and γ (analgus t MAP estimatin)

29 Example Inference Prbability Tpics

30 Example Inference human evlutin disease cmputer genme evlutinary hst mdels dna species bacteria infrmatin genetic rganisms diseases data genes life resistance cmputers sequence rigin bacterial system gene bilgy new netwrk mlecular grups strains systems sequencing phylgenetic cntrl mdel map living infectius parallel infrmatin diversity malaria methds genetics grup parasite netwrks mapping new parasites sftware prject tw united new sequences cmmn tuberculsis simulatins

31 Example Inference

32 Example Inference prblem mdel selectin species prblems rate male frest mathematical cnstant males eclgy number distributin females fish new time sex eclgical mathematics number species cnservatin university size female diversity tw values evlutin ppulatin first value ppulatins natural numbers average ppulatin ecsystems wrk rates sexual ppulatins time data behavir endangered mathematicians density evlutinary trpical chas measured genetic frests chatic mdels reprductive ecsystem

33 Perfrmance Metric: Perplexity Nematde abstracts Assciated Press Smthed Unigram Smthed Mixt. Unigrams LDA Fld in plsi Smthed Unigram Smthed Mixt. Unigrams LDA Fld in plsi Perplexity Number f Tpics Number f Tpics perplexity = exp P d lg p(w d) P d N d Marginal likelihd (evidence) f held ut dcuments

34 Extensins f LDA EM inference (PLSA/PLSI) yields similar results t Variatinal inference r MAP inference (LDA) n mst data Reasn fr ppularity f LDA: can be embedded in mre cmplicated mdels

35 Extensins: Supervised LDA d Z d,n W d,n N k K Y d D η, σ 2 1 Draw tpic prprtins Dir( ). 2 Fr each wrd Draw tpic assignment z n Mult( ). Draw wrd w n z n, 1:K Mult( zn ). 3 Draw respnse variable y z 1:N,, 2 N > z, 2, where z =(1/N) P N n=1 z n.

36 Extensins: Supervised LDA least prblem unfrtunately suppsed wrse flat dull bad guys watchable its nt ne mvie mre has than films directr will characters awful featuring rutine dry ffered charlie paris his their character many while perfrmance between bth mtin simple perfect fascinating pwer cmplex have like yu was just sme ut nt abut mvie all wuld they its ne frm there which wh much what hwever cinematgraphy screenplay perfrmances pictures effective picture

37 Extensins: Crrelated Tpic Mdel k d Z d,n W d,n N D K µ Ncnjugate prir n tpic prprtins Estimate a cvariance matrix Σ that parameterizes crrelatins between tpics in a dcument

38 Extensins: Dynamic Tpic Mdels Dynamic tpic mdels (Blei and Lafferty, 2006) Inaugural addresses My fellw citizens: I stand here tday humbled by the task befre us, grateful fr the trust yu have bestwed, mindful f the sacrifices brne by ur ancestrs... AMONG the vicissitudes incident t life n event culd have filled me with greater anxieties than that f which the ntificatin was transmitted by yur rder... Trackthat changes distributins LDA assumes the rderinfwrd dcuments des nt matter. assciated withthat a tpic ver time. Nt apprpriate fr crpra span hundreds f years We may want t track hw language changes ver time.

39 Extensins: Dynamic Tpic Mdels d d d Z d,n Z d,n Z d,n W d,n W d,n W d,n N D N D N D... K β k,1 β k,2 β k,t

40 Extensins: Dynamic Tpic Mdels 1880 electric machine pwer engine steam tw machines irn battery wire 1890 electric pwer cmpany steam electrical machine tw system mtr engine 1900 apparatus steam pwer engine engineering water cnstructin engineer rm feet 1910 air water engineering apparatus rm labratry engineer made gas tube 1920 apparatus tube air pressure water glass gas made labratry mercury 1930 tube apparatus glass air mercury labratry pressure made gas small 1940 air tube apparatus glass labratry rubber pressure small mercury gas 1950 tube apparatus glass air chamber instrument small labratry pressure rubber 1960 tube system temperature air heat chamber pwer high instrument cntrl 1970 air heat pwer system temperature chamber high flw tube design 1980 high pwer design heat system systems devices instruments cntrl large 1990 materials high pwer current applicatins technlgy devices design device heat 2000 devices device materials current gate high light silicn material technlgy

41 Extensins: Dynamic Tpic Mdels "Theretical Physics" "Neurscience" FORCE RELATIVITY LASER NERVE OXYGEN NEURON

42 Extensins: Ideal Pint Tpic Mdels 2 d 2 u d Z dn W dn N A d,b d V ud D X u U k K Bill cntent (tpic mdel) Bill sentiment variables Observed vtes Legislatr ideal pints

43 Extensins: Ideal Pint Tpic Mdels tax credit,budget authrity,energy,utlays,tax cunty,eligible,ballt,electin,jurisdictin bank,transfer,requires,hlding cmpany,industrial husing,mrtgage,lan,family,recipient energy,fuel,standard,administratr,lamp student,lan,institutin,lender,schl medicare,medicaid,child,chip,cverage defense,iraq,transfer,expense,chapter business,administratr,bills,business cncern,lan transprtatin,rail,railrad,passenger,hmeland security cver,bills,bridge,transactin,fllwing bills,tax,subparagraph,lss,taxable lss,crp,prducer,agriculture,trade head,start,child,technlgy,award cmputer,alien,bills,user,cllectin science,directr,technlgy,mathematics,bills cast guard,vessel,space,administratr,requires child,center,pisn,victim,abuse land,site,bills,interir,river energy,bills,price,cmmdity,market surveillance,directr,curt,electrnic,fld child,fire,attrney,internet,bills drug,pediatric,prduct,device,medical human,vietnam,united natins,call,peple bills,iran,fficial,cmpany,sudan cin,inspectr,designee,autmbile,lebann prducer,eligible,crp,farm,subparagraph peple,wman,american,natin,schl veteran,veterans,bills,care,injury dd,defense,defense and apprpriatin,military,subtitle