Data Mining and MapReduce. Adapted from Lectures by Prabhakar Raghavan (Yahoo and Stanford) and Christopher Manning (Stanford)

Transcription

1 Data Mining and MapRedue Adapted from Letures by Prabhakar Raghavan Yahoo and Stanford and Christopher Manning Stanford 1

2 2 Overview Text Classifiation K-Means Classifiation The Naïve Bayes algorithm

3 3 Relevane feedbak revisited In relevane feedbak, the user marks a number of douments as relevant/nonrelevant, that an be used to improve searh results. Suppose we ust tried to learn a filter for nonrelevant douments This is an instane of a text lassifiation problem: Two lasses : relevant, nonrelevant For eah doument, deide whether it is relevant or nonrelevant The notion of lassifiation is very general and has many appliations within and beyond IR.

4 4 Standing queries The path from information retrieval to text lassifiation: You have an information need, say: Unrest in the Niger delta region You want to rerun an appropriate query periodially to find new news items on this topi I.e., it s lassifiation not ranking Suh queries are alled standing queries Long used by information professionals A modern mass instantiation is Google Alerts

5 Spam filtering: Another text 5 From: "" <takworlld@hotmail.om> Subet: real estate is the only way... gem oalvgkay Anyone an buy real estate with no money down Stop paying rent TODAY! lassifiation task There is no need to spend hundreds or even thousands for similar ourses I am 22 years old and I have already purhased 6 properties using the methods outlined in this truly INCREDIBLE ebook. Change your life NOW! ================================================= Clik Below to order: =================================================

6 6 Categorization/Classifiation Given: A desription of an instane, x X, where X is the instane language or instane spae. Issue: how to represent text douments. A fixed set of lasses: C = { 1, 2,, J } Determine: The ategory of x: xc, where x is a lassifiation funtion whose domain is X and whose range is C. We want to know how to build lassifiation funtions lassifiers.

7 7 More Text Classifiation Examples: Many searh engine funtionalities use lassifiation Assign labels to eah doument or web-page: Labels are most often topis suh as Yahoo-ategories e.g., "finane," "sports," "news>world>asia>business" Labels may be genres e.g., "editorials" "movie-reviews" "news Labels may be opinion on a person/produt e.g., like, hate, neutral Labels may be domain-speifi e.g., "interesting-to-me" : "not-interesting-to-me e.g., ontains adult language : doesn t e.g., language identifiation: English, Frenh, Chinese, e.g., searh vertial: about Linux versus not e.g., link spam : not link spam

8 8 Classifiation Methods 1 Manual lassifiation Used by Yahoo! originally; now downplayed, Looksmart, about.om, ODP, PubMed Very aurate when ob is done by experts Consistent when the problem size and team is small Diffiult and expensive to sale Means we need automati lassifiation methods for big problems

9 Classifiation Methods 2 Supervised learning of a doument-label assignment funtion Many systems partly rely on mahine learning Autonomy, MSN, Verity, Enkata, Yahoo!, k-nearest Neighbors simple, powerful Naive Bayes simple, ommon method Support-vetor mahines new, more powerful plus many other methods No free lunh: requires hand-lassified training data But data an be built up and refined by amateurs Note that many ommerial systems use a mixture of methods 9

10 Clustering k-means Basi and fundamental Original Algorithm 1. Pik k initial enter points 2. Iterate until onverge 1. Assign eah point with the nearest enter 2. Calulate new enters Easy to parallellize

11 K-Means Clustering 11

12 12 How Does K-means partition? When K entroids are set/fixed, they partition the whole data spae into K mutually exlusive subspaes to form a partition. A partition amounts to a Voronoi Diagram Changing positions of entroids leads to a new partitioning.

13 K-means Demo 1. User set up the number of lusters they d like. e.g. k=5 13

14 K-means Demo 1. User set up the number of lusters they d like. e.g. K=5 2. Randomly guess K luster Center loations 14

15 K-means Demo 1. User set up the number of lusters they d like. e.g. K=5 2. Randomly guess K luster Center loations 3. Eah data point finds out whih Center it s losest to. Thus eah Center owns a set of data points 15

16 K-means Demo 1. User set up the number of lusters they d like. e.g. K=5 2. Randomly guess K luster entre loations 3. Eah data point finds out whih entre it s losest to. Thus eah Center owns a set of data points 4. Eah entre finds the entroid of the points it owns 16

17 K-means Demo 1. User set up the number of lusters they d like. e.g. K=5 2. Randomly guess K luster entre loations 3. Eah data point finds out whih entre it s losest to. Thus eah entre owns a set of data points 4. Eah entre finds the entroid of the points it owns 5. and umps there 17

18 K-means Demo 1. User set up the number of lusters they d like. e.g. K=5 2. Randomly guess K luster entre loations 3. Eah data point finds out whih entre it s losest to. Thus eah entre owns a set of data points 4. Eah entre finds the entroid of the points it owns 5. and umps there 6. Repeat until terminated! 18

19 K-Means Clustering with MapRedue Mapper_i-1 Eah Mapper loads a set of data samples, and assign eah sample to a nearest entroid Reduer_i-1 Mapper_i Reduer_i 4 Mapper_i+1 Eah Mapper needs to keep a opy of entroids Reduer_i How to set the initial entroids is very important! Usually we set the entroids using Canopy Clustering. [MCallum, Nigam and Ungar: "Effiient Clustering of High Dimensional Data Sets with Appliation to Referene Mathing", SIGKDD 2000]

20 Clustering k-means a shared file ontains entroid points map 1. for eah point, find the nearest enter 2. generate <key,value> pair key: enter id value: urrent point s oordinate redue 1. ollet all points belonging to the same luster they have the same key value 2. alulate the average new enter iterate

21 Probabilisti relevane feedbak Rather than re-weighting in a vetor spae If user has told us some relevant and some irrelevant douments, then we an proeed to build a probabilisti lassifier, suh as the Naive Bayes model: Pt k R = D rk / D r Pt k NR = D nrk / D nr t k is a term; D r is the set of known relevant douments; D rk is the subset that ontain t k ; D nr is the set of known irrelevant douments; D nrk is the subset that ontain t k.

22 Reall a few probability basis For events a and b: Bayes Rule Odds: a a x x p x b p a p a b p b p a p a b p b a p a p a b p b p b a p a p a b p b p b a p b a p b a p,, 1 a p a p a p a p a O Posterior Prior

23 Bayesian Methods Learning and lassifiation methods based on probability theory. Bayes theorem plays a ritial role in probabilisti learning and lassifiation. Build a generative model that approximates how data is produed. Uses prior probability of eah ategory given no information about an item. Categorization produes a posterior probability distribution over the possible ategories given a desription of an item and prior probabilities.

24 24 Bayes Rule P C, D P C D P D P D C P C PC D PD CPC PD

25 25 The Naïve Bayes Classifier Flu X 1 X 2 X 3 X 4 X 5 runnynose sinus ough fever musle-ahe Conditional Independene Assumption: Features term presene are independent of eah other given the lass: P 5 X1,, X5 C P X1 C P X 2 C P X C This model is appropriate for binary variables Multivariate Bernoulli model

26 Naive Bayes Classifiers Task: Classify a new instane D based on a tuple of attribute values into one of the lasses C 26 x n x x D,, 2, 1,,, argmax 2 1 n C MAP x x x P,,,,,, argmax n n C x x x P P x x x P,,, argmax 2 1 n C P x x x P MAP = Maximum Aposteriori Probability

27 Naïve Bayes Classifier: 27 Naïve Bayes Assumption P Can be estimated from the frequeny of lasses in the training examples. Px 1,x 2,,x n O X n C parameters Could only be estimated if a very, very large number of training examples was available. Naïve Bayes Conditional Independene Assumption: Assume that the probability of observing the onuntion of attributes is equal to the produt of the individual probabilities Px i.

28 28 Example Example: Play Tennis

29 29 Example Learning Phase Outlook Play=Yes Play=No Sunny 2/9 3/5 Overast 4/9 0/5 Rain 3/9 2/5 Temperature Play=Yes Play=No Hot 2/9 2/5 Mild 4/9 2/5 Cool 3/9 1/5 Humidity Play=Yes Play=No High 3/9 4/5 Normal 6/9 1/5 Wind Play=Yes Play=No Strong 3/9 3/5 Weak 6/9 2/5 PPlay=Yes = 9/14 PPlay=No = 5/14

30 30 Example Test Phase Given a new instane, predit its label x =Outlook=Sunny, Temperature=Cool, Humidity=High, Wind=Strong Look up tables ahieved in the learning phrase POutlook=Sunny Play=Yes = 2/9 PTemperature=Cool Play=Yes = 3/9 PHuminity=High Play=Yes = 3/9 PWind=Strong Play=Yes = 3/9 PPlay=Yes = 9/14 POutlook=Sunny Play=No = 3/5 PTemperature=Cool Play==No = 1/5 PHuminity=High Play=No = 4/5 PWind=Strong Play=No = 3/5 PPlay=No = 5/14 Deision making with the MAP rule PYes x : [PSunny YesPCool YesPHigh YesPStrong Yes]PPlay=Yes = PNo x : [PSunny No PCool NoPHigh NoPStrong No]PPlay=No = Given the fat PYes x < PNo x, we label x to be No.

31 Learning the Model First attempt: maximum likelihood estimates simply use the frequenies in the data 31, ˆ i i i C N C x X N x P C X 1 X 2 X 5 X 3 X 4 X 6 N C N P ˆ

32 Problem with Max Likelihood Flu X 1 X 2 X 3 X 4 X 5 runnynose sinus ough fever musle-ahe P 5 X1,, X5 C P X1 C P X 2 C P X C What if we have seen no training ases where patient had no flu and musle ahes? ˆ N X t, C nf P X 5 5 t C nf N C nf Zero probabilities annot be onditioned away, no matter the other evidene! arg max P ˆ Pˆ x i i 0

33 MapRedue Approah: Simple Method X 1 X 2 X 3 X 4 X 5 sinus Flu ough Flu fever Flu Runnynose Flu I 1 : Runnynose,sinus,ough,fever,musle-ahe,Flu I m : Runnynose,sinus,ough, fever, musle-ahe, Flu musle-ahe Flu Flu X 1 X 2 X 3 X 4 X 5 sinus Fluough Flufever Flu Runnynose Flu MAP musle-ahe Flu Flu REDUCE X 1 X 2 X 3 X 4 X 5 PRunnynose Flu Psinus Flu Pough Flu Pfever Flu Pmusle-ahe Flu PFlu

34 Mahine Learning Algorithm Transformation Naïve Bayes Classifiation more omplex arg max ]... [ arg max ] [ arg max... arg max i V v NB n V v MAP n n V v MAP n V v MAP v a P v P v v P v a a P v a a P v P v a a P v a a v P v map redue

35 Mahine Learning Algorithm Transformation to MapRedue Solution Find statistis alulation part Distribute alulations on data using map Gather and refine all statistis in redue

36 36

37 Exerise Estimate parameters of Naive Bayes lassifier Classify test doument 37

38 Example: Parameter estimates The denominators are and beause the lengths of text and are 8 and 3, respetively, and beause the onstant B is 6 as the voabulary onsists of six terms. 38

39 Example: Classifiation Thus, the lassifier assigns the test doument to = China. The reason for this lassifiation deision is that the three ourrenes of the positive indiator CHINESE in d 5 outweigh the ourrenes of the two negative indiators JAPAN and TOKYO. 39

40 Example: Sensors Raining Reality Sunny P+,+,r = 3/8 P-,-,r = 1/8 P+,+,s = 1/8 P-,-,s = 3/8 NB Model Raining? M1 M2 NB FACTORS: Ps = 1/2 P+ s = 1/4 P+ r = 3/4 PREDICTIONS: Pr,+,+ = ½¾¾ Ps,+,+ = ½¼¼ Pr +,+ = 9/10 Ps +,+ = 1/10 40

41 Prasad L13NaiveBayesClassify 41

42 42 Smoothing to Avoid Overfitting Pˆ x i N X i N C x i, C k 1 # of values of X i Prasad L13NaiveBayesClassify

43 Smoothing to Avoid Overfitting k C N C x X N x P i i i 1, ˆ Somewhat more subtle version Prasad L13NaiveBayesClassify 43 # of values of X i m C N mp C x X N x P k i k i i k i, ˆ,,, overall fration in data where X i =x i,k extent of smoothing

44 Using Multinomial Naive Bayes Classifiers to Classify Text: Basi method 48 Attributes are text positions, values are words. NB argmax C argmax C P P P x Still too many possibilities P x P x Assume that lassifiation is independent of the positions of the words Use same parameters for eah position i 1 Result is bag of words model over tokens not types i "our" n "text" Prasad L13NaiveBayesClassify

45 49 Naïve Bayes: Learning Algorithm From training orpus, extrat Voabulary Calulate required P and Px k terms For eah in C do dos subset of douments for whih the target lass is P dos total# douments Text single doument ontaining all dos for eah word x k in Voabulary n k number of ourrenes of x k in Text P x k nk n Voabulary Prasad L13NaiveBayesClassify

46 50 Naïve Bayes: Classifying positions all word positions in urrent doument whih ontain tokens found in Voabulary Return NB, where NB argmax C P P x ipositions i Prasad L13NaiveBayesClassify

47 51 Naive Bayes: Time Complexity Training Time: O D L d + C V where L d is the average length of a doument in D. Assumes V and all D i, n i, and n i pre-omputed in O D L d time during one pass through all of the data. Why? Generally ust O D L d sine usually C V < D L d Test Time: O C L t where L t is the average length of a test doument. Very effiient overall, linearly proportional to the time needed to ust read in all the data. Plus, robust in pratie Prasad L13NaiveBayesClassify

48 52 Underflow Prevention: log spae Multiplying lots of probabilities, whih are between 0 and 1, an result in floating-point underflow. Sine logxy = logx + logy, it is better to perform all omputations by summing logs of probabilities rather than multiplying probabilities. Class with highest final un-normalized log probability sore is still the most probable. argmax Note NB that model is now ust max of sum of weights C log P ipositions log P x i Prasad L13NaiveBayesClassify

49 53 Note: Two Models Model 1: Multivariate Bernoulli One feature X w for eah word in ditionary X w = true in doument d if w appears in d Naive Bayes assumption: Given the doument s topi, appearane of one word in the doument tells us nothing about hanes that another word appears This is the model used in the binary independene model in lassi probabilisti relevane feedbak in hand-lassified data Prasad L13NaiveBayesClassify

50 54 Two Models Model 2: Multinomial = Class onditional unigram One feature X i for eah word pos in doument feature s values are all words in ditionary Value of X i is the word in position i Naïve Bayes assumption: Given the doument s topi, word in one position in the doument tells us nothing about words in other positions Seond assumption: Word appearane does not depend on position P X w P X w i for all positions i,, word w, and lass Prasad

51 55 Parameter estimation Multivariate Bernoulli model: Pˆ X t w fration of douments of topi in whih word w appears Multinomial model: Pˆ X w i fration of times in whih word w appears aross all douments of topi Can reate a mega-doument for topi by onatenating all douments on this topi Use frequeny of w in mega-doument Prasad L13NaiveBayesClassify

52 56 Classifiation Multinomial vs Multivariate Bernoulli? Multinomial model is almost always more effetive in text appliations! See IIR setions 13.2 and 13.3 for worked examples with eah model Prasad L13NaiveBayesClassify

53 Feature Seletion: Why? Prasad Text olletions have a large number of features 10,000 1,000,000 unique words and more Feature Seletion Makes using a partiular lassifier feasible Some lassifiers an t deal with 100,000 of features Redues training time Training time for some methods is quadrati or worse in the number of features Can improve generalization performane Eliminates noise features Avoids overfitting L13NaiveBayesClassify

54 Feature seletion: how? Two ideas: Hypothesis testing statistis: Are we onfident that the value of one ategorial variable is assoiated with the value of another Chi-square test 2 Information theory: How muh information does the value of one ategorial variable give you about the value of another Mutual information MI They re similar, but 2 measures onfidene in assoiation, based on available statistis, while MI measures extent of assoiation assuming perfet knowledge of probabilities Prasad L13NaiveBayesClassify

55 2 statisti CHI 2 is interested in f o f e 2 /f e summed over all table entries: is the observed number what you d expet given the marginals? , a / EO.2./ / / / p0. The null hypothesis is reeted with onfidene.999, sine 12.9 > the value for.999 onfidene. Term = aguar Term aguar expeted: f e Class = auto Class auto observed: f o

56 2 statisti CHI There is a simpler formula for 2x2 2 : A = #t, B = #t, C = # t, D = # t, Yields 1.29? N = A + B + C + D Value for omplete independene of term and ategory?

57 Feature seletion via Mutual Information In training set, hoose k words whih best disriminate give most info. on the ategories. The Mutual Information between a word, lass is: I w, e w p e, e log w w { 0,1} e{ 0,1} p ew p e For eah word w and eah ategory p e, e Prasad L13NaiveBayesClassify

58 62 Feature seletion via MI ontd. For eah ategory we build a list of k most disriminating terms. For example on 20 Newsgroups: si.eletronis: iruit, voltage, amp, ground, opy, battery, eletronis, ooling, re.autos: ar, ars, engine, ford, dealer, mustang, oil, ollision, autos, tires, toyota, Greedy: does not aount for orrelations between terms Prasad L13NaiveBayesClassify

59 63 Feature Seletion Mutual Information Clear information-theoreti interpretation May selet rare uninformative terms Chi-square Statistial foundation May selet very slightly informative frequent terms that are not very useful for lassifiation Just use the ommonest terms? No partiular foundation In pratie, this is often 90% as good Prasad L13NaiveBayesClassify

60 64 Feature seletion for NB In general, feature seletion is neessary for multivariate Bernoulli NB. Otherwise, you suffer from noise, multi-ounting Feature seletion really means something different for multinomial NB. It means ditionary trunation. This feature seletion normally isn t needed for multinomial NB, but may help a fration with quantities that are badly estimated Prasad L13NaiveBayesClassify

61 WebKB Experiment 1998 Classify webpages from CS departments into: student, faulty, ourse,proet Train on ~5,000 hand-labeled web pages Cornell, Washington, U.Texas, Wisonsin Crawl and lassify a new site CMU Results: Student Faulty Person Proet Course Departmt Extrated Corret Auray: 72% 42% 79% 73% 89% 100%

62 67 NB Model Comparison: WebKB Prasad

63 68 Prasad L13NaiveBayesClassify

64 Naïve Bayes on spam Prasad L13NaiveBayesClassify

65 71 Violation of NB Assumptions Conditional independene Positional independene Examples? Prasad L13NaiveBayesClassify

66 73 Naive Bayes is Not So Naive Naïve Bayes: First and Seond plae in KDD-CUP 97 ompetition, among 16 then state of the art algorithms Goal: Finanial servies industry diret mail response predition model: Predit if the reipient of mail will atually respond to the advertisement 750,000 reords. Robust to Irrelevant Features Irrelevant Features anel eah other without affeting results Instead Deision Trees an heavily suffer from this. Very good in domains with many equally important features Deision Trees suffer from fragmentation in suh ases espeially if little data A good dependable baseline for text lassifiation but not the best! Optimal if the Independene Assumptions hold: If assumed independene is orret, then it is the Bayes Optimal Classifier for problem Very Fast: Learning with one pass of ounting over the data; testing linear in the number of attributes, and doument olletion size Low Storage requirements Prasad L13NaiveBayesClassify