Data Mining Techniques

Transcription

1 Data Mining Techniques CS Sectin 3 - Fall 2016 Lecture 11 Jan-Willem van de Meent (credit: Yijun Zha, Dave Blei)

2 PROJECT GUIDELINES (updated)

3 Prject Gals Select a dataset / predictin prblem Perfrm explratry analysis and preprcesssing Apply ne r mre algrithms Critically evaluate results Submit a reprt and present prject

4 Prpsals Due: 28 Octber Presentatin:10+5 mins Prpsal: 1-2 pages Describe Dataset Predictin task Prpsed methds

5 Presentatin and Reprt Due: 2 December Presentatin 20 mins + 10 discussin Reprt 8-10 pages, 11 pts Cde

6 Presentatin and Reprt Due: 2 December Presentatin 20 mins + 10 discussin Reprt 8-10 pages, 11 pts Cde

7 Grading Prpsal: 15% Prblem and Results: 20% Data and Cde: 15% Reprt: 35% Presentatin: 15%

8 Grading Prblem and Results: 20% Nvelty f task Own dataset vs UCI dataset Number f algrithms tested Nvelty f algrithms

9 Grading Data and Cde: 15% Dcumentatin and Readability TAs shuld be able t run cde Reprducibility (can figures and tables be generated by running cde?)

10 Grading Reprt: 35% Explratry analysis f data Explain hw prperties f data relate t chice f algrithm Descriptin f algrithms and methdlgy Discussin f results Which methds wrk well, which d nt, and why? Cmparisn t state f art?

11 Example: Minimum Viable Prject Get 2-3 datasets frm UCI repsitry Figure ut what pre-prcessing (if any) is needed Run every applicable algrithm in scikit learn Explain which algrithms wrk well n which datasets and why

12 Example: Mre Ambitius Prjects Find a new dataset r define a nvel task (i.e. nt classificatin r clustering) Attack a prblem frm a Kaggle cmpetitin Implement a recently published methd (talk t me fr suggestins)

13 Hmewrk Updates HW3 nw due n 2 Nvember (after midterm and prpsals) Remved HW5 t give mre time t wrk n prjects

14 MIDTERM REVIEW

15 List f Tpics fr Midterm Everything up until last Friday (expect final t emphasize later tpics) Open bk, fcus n understanding

16 BINOMIAL MIXTURES

17 Mixture f Binmials Suppse we have tw cins A and B (weighted). We want t estimate the bias f the tw cins. i.e. p A (head) =µ A p B (head) =µ B Pick a cin at randm (simplified versin, a equal mixture) Flip 10 times and recrd H and T repeat the prcess until we have a gd size f training data

18 Mixture f Binmials

19 Gaussian Mixture Mdel Generative Mdel Expectatin Maximizatin Initialize θ Repeat until cnvergence 1. Expectatin Step 2. Maximizatin Step

20 Binmial Mixture Mdel Generative Mdel Expectatin Maximizatin Initialize θ Repeat until cnvergence 1. Expectatin Step 2. Maximizatin Step

21 Binmial Mixture Mdel Generative Mdel Expectatin Maximizatin Initialize θ Repeat until cnvergence 1. Expectatin Step 2. Maximizatin Step

22 TOPIC MODELS Brrwing frm: David Blei (Clumbia)

23 Review: Naive Bayes Features: Wrds in x = n a aardvark aardwlf. buy. zygmurgy Generative Mdel Maximum Likelihd Labels: Spam r nt Spam

24 Review: Naive Bayes Features: Wrds in x = n a aardvark aardwlf. buy. zygmurgy Generative Mdel (with prir) Maximum Likelihd Psterir Mean fr Parameters Labels: Spam r nt Spam

25 Mixtures f Dcuments Observatins: Bag f Wrds x = n a aardvark aardwlf. buy. zygmurgy Clusters: Types f Dcuments

26 Mixtures f Dcuments Observatins: Bag f Wrds x = n a aardvark aardwlf. buy. zygmurgy Clusters: Types f Dcuments Generative Mdel (with prir) Maximum Likelihd Hw shuld we mdify the generative mdel?

27 Mixtures f Dcuments Observatins: Bag f Wrds x = n a aardvark aardwlf. buy. zygmurgy Generative Mdel (with prir) Clusters: Types f Dcuments

28 Tpic Mdeling Tpics gene 0.04 dna 0.02 genetic 0.01.,, Dcuments Tpic prprtins and assignments life 0.02 evlve 0.01 rganism 0.01.,, brain 0.04 neurn 0.02 nerve data 0.02 number 0.02 cmputer 0.01.,, Naive Bayes: Dcuments belng a class Tpic Mdels: Wrds belng t a class

29 Latent Dirichlet Allcatin Prprtins parameter Per-wrd tpic assignment Per-dcument tpic prprtins Observed wrd Tpics Tpic parameter d Z d,n W d,n N k D K η

30 PLSI/PLSA: EM fr LDA Generative Mdel (n prirs) Expectatin Step Maximizatin Step

31 Variatinal Inference fr LDA (sketch) Generative Mdel : d Z d,n W d,n N k D K η Variatinal Apprximatin "d!d N %d,n!d,n #k $k D K

32 Variatinal Inference fr LDA (sketch) Generative Mdel : d Z d,n W d,n N k D K η Variatinal Apprximatin "d!d N %d,n!d,n #k $k D K

33 Variatinal Inference fr LDA (sketch) One iteratin f mean field variatinal inference fr LDA (1) Fr each tpic k and term v: D N (t+1) k,v = + 1(w d,n = v) (t) n,k. (2) Fr each dcument d: (a) Update d : d=1 n=1 (t+1) d,k = k + N n=1 (t) d,n,k. (b) Fr each wrd n, update d,n : 0) (t+1) d,n,k exp ( (t+1) d,k ) + ( (t+1) k,w n ) ( V v=1 (t+1) k,v ) where is the digamma functin, the first derivative f the lg functin.,

34 Example Inference Prbability Tpics

35 Example Inference human evlutin disease cmputer genme evlutinary hst mdels dna species bacteria infrmatin genetic rganisms diseases data genes life resistance cmputers sequence rigin bacterial system gene bilgy new netwrk mlecular grups strains systems sequencing phylgenetic cntrl mdel map living infectius parallel infrmatin diversity malaria methds genetics grup parasite netwrks mapping new parasites sftware prject tw united new sequences cmmn tuberculsis simulatins

36 Example Inference

37 Example Inference prblem mdel selectin species prblems rate male frest mathematical cnstant males eclgy number distributin females fish new time sex eclgical mathematics number species cnservatin university size female diversity tw values evlutin ppulatin first value ppulatins natural numbers average ppulatin ecsystems wrk rates sexual ppulatins time data behavir endangered mathematicians density evlutinary trpical chas measured genetic frests chatic mdels reprductive ecsystem

38 Perfrmance Metric: Perplexity Nematde abstracts Assciated Press Smthed Unigram Smthed Mixt. Unigrams LDA Fld in plsi Smthed Unigram Smthed Mixt. Unigrams LDA Fld in plsi Perplexity Number f Tpics Number f Tpics perplexity = exp P d lg p(w d) P d N d Marginal likelihd (evidence) f held ut dcuments

39 Extensins f LDA EM inference (PLSA/PLSI) yields similar results t Variatinal inference (LDA) n mst data Reasn fr ppularity f LDA: can be embedded in mre cmplicated mdels

40 Extensins: Crrelated Tpic Mdel k d Z d,n W d,n N D K µ Ncnjugate prir n tpic prprtins Estimate a cvariance matrix Σ that parameterizes crrelatins between tpics in a dcument

41 Extensins: Dynamic Tpic Mdels Dynamic tpic mdels (Blei and Lafferty, 2006) Inaugural addresses My fellw citizens: I stand here tday humbled by the task befre us, grateful fr the trust yu have bestwed, mindful f the sacrifices brne by ur ancestrs... AMONG the vicissitudes incident t life n event culd have filled me with greater anxieties than that f which the ntificatin was transmitted by yur rder... Trackthat changes distributins LDA assumes the rderinfwrd dcuments des nt matter. assciated withthat a tpic ver time. Nt apprpriate fr crpra span hundreds f years We may want t track hw language changes ver time.

42 Extensins: Dynamic Tpic Mdels d d d Z d,n Z d,n Z d,n W d,n W d,n W d,n N D N D N D... K β k,1 β k,2 β k,t

43 Extensins: Dynamic Tpic Mdels d d d Z d,n Z d,n Z d,n W d,n W d,n W d,n N D N D N D... K β k,1 β k,2 β k,t

44 Extensins: Dynamic Tpic Mdels 1880 electric machine pwer engine steam tw machines irn battery wire 1890 electric pwer cmpany steam electrical machine tw system mtr engine 1900 apparatus steam pwer engine engineering water cnstructin engineer rm feet 1910 air water engineering apparatus rm labratry engineer made gas tube 1920 apparatus tube air pressure water glass gas made labratry mercury 1930 tube apparatus glass air mercury labratry pressure made gas small 1940 air tube apparatus glass labratry rubber pressure small mercury gas 1950 tube apparatus glass air chamber instrument small labratry pressure rubber 1960 tube system temperature air heat chamber pwer high instrument cntrl 1970 air heat pwer system temperature chamber high flw tube design 1980 high pwer design heat system systems devices instruments cntrl large 1990 materials high pwer current applicatins technlgy devices design device heat 2000 devices device materials current gate high light silicn material technlgy

45 Extensins: Dynamic Tpic Mdels "Theretical Physics" "Neurscience" FORCE RELATIVITY LASER NERVE OXYGEN NEURON

46 Extensins: Supervised LDA d Z d,n W d,n N k K Y d D η, σ 2 1 Draw tpic prprtins Dir( ). 2 Fr each wrd Draw tpic assignment z n Mult( ). Draw wrd w n z n, 1:K Mult( zn ). 3 Draw respnse variable y z 1:N,, 2 N > z, 2, where z =(1/N) P N n=1 z n.

47 Extensins: Supervised LDA least prblem unfrtunately suppsed wrse flat dull bad guys watchable its nt ne mvie mre has than films directr will characters awful featuring rutine dry ffered charlie paris his their character many while perfrmance between bth mtin simple perfect fascinating pwer cmplex have like yu was just sme ut nt abut mvie all wuld they its ne frm there which wh much what hwever cinematgraphy screenplay perfrmances pictures effective picture

48 Extensins: Ideal Pint Tpic Mdels 2 d 2 u d Z dn W dn N A d,b d V ud D X u U k K Bill cntent (tpic mdel) Bill sentiment variables Observed vtes Legislatr ideal pints

49 Extensins: Ideal Pint Tpic Mdels tax credit,budget authrity,energy,utlays,tax cunty,eligible,ballt,electin,jurisdictin bank,transfer,requires,hlding cmpany,industrial husing,mrtgage,lan,family,recipient energy,fuel,standard,administratr,lamp student,lan,institutin,lender,schl medicare,medicaid,child,chip,cverage defense,iraq,transfer,expense,chapter business,administratr,bills,business cncern,lan transprtatin,rail,railrad,passenger,hmeland security cver,bills,bridge,transactin,fllwing bills,tax,subparagraph,lss,taxable lss,crp,prducer,agriculture,trade head,start,child,technlgy,award cmputer,alien,bills,user,cllectin science,directr,technlgy,mathematics,bills cast guard,vessel,space,administratr,requires child,center,pisn,victim,abuse land,site,bills,interir,river energy,bills,price,cmmdity,market surveillance,directr,curt,electrnic,fld child,fire,attrney,internet,bills drug,pediatric,prduct,device,medical human,vietnam,united natins,call,peple bills,iran,fficial,cmpany,sudan cin,inspectr,designee,autmbile,lebann prducer,eligible,crp,farm,subparagraph peple,wman,american,natin,schl veteran,veterans,bills,care,injury dd,defense,defense and apprpriatin,military,subtitle