CS 331: Artificial Intelligence Naïve Bayes. Naïve Bayes

Transcription

1 CS 33: Artificial Intelligence Naïe Bayes Thanks to Andrew Moore for soe corse aterial Naïe Bayes A special type of Bayesian network Makes a conditional independence assption Typically sed for classification 2

2 Classification Sppose yo are trying to classify sitations that deterine whether or not Canas will be down. o e coe p with the following list of ariables which are all Boolean: Monday Assn Grades Win Is a Monday CS33 assignent de CS33 instrctor needs to enter grades The Beaers won the football gae We also hae a Boolean ariable called CD which stands for Canas down 3 Classification These are called featres or attribtes This is called the class ariable becase we re trying to classify it Monday Assn Grades Win CD tre tre tre false tre false tre tre tre false tre false false false false false tre tre false tre tre tre tre false tre false false tre false tre tre tre false tre false These entries in the CD coln are called class labels 4 2

3 Monday Assn Grades Win CD tre tre tre false tre false tre tre tre false tre false false false false false tre false false tre tre tre tre false tre false false tre false tre tre tre false tre false Monday Assn Grades Win tre tre tre tre false tre tre false Classification o create a dataset ot of yor past experience. This is called training data. o now hae 2 new sitations and yo wold like to if Canas will go down. This is called test data. 5 Naïe Bayes Strctre CD M A G W Notice the conditional independence assption: The featres are conditionally independent gien the class ariable. 6 3

4 Naïe Bayes araeters CD =? CD M A G W M CD =? A CD =? G CD =? W CD =? How do yo get these paraeters fro the training data? 7 Naïe Bayes araeters CD M A G W CD CD false # of records in training data with CD = false / # of records in training data tre # of records in training data with CD = tre / # of records in training data 8 4

5 Naïe Bayes araeters CD M A G W M CD M CD false false # of records with M = false and CD = false / # of records with CD = false false tre # of records with M = false and CD = tre / # of records with CD = tre tre false # of records with M = tre and CD = false / # of records with CD = false tre tre # of records with M = tre and CD = tre / # of records with CD = tre Inference in Naïe Bayes CD M, A, G, W M, A, G, W CD CD M, A, G, W M, A, G, W CD CD By Bayes Rle Treat denoinator as constant CD M CD A CD G CD W Fro conditional independence CD 0 5

6 rediction Sppose yo are now in a day when M=tre, A=tre, G=tre, W=tre. o need to if CD=tre or CD=false. We will se the notation that CD=tre is eqialent to cd and CD=false is eqialent to cd. o need to copare: rediction cd, a, g, w = α cd cd a cd g cd w cd cd, a, g, w = α cd cd a cd g cd w cd Whicheer probability is the bigger of the two aboe, that is yor ion for CD Becase yo take the ax of the two probabilities aboe, yo can ignore α since it is the sae in both 2 6

7 The General Case Estiate = as fraction of records with = 2. Estiate i = = as fraction of = records that also hae =. 3. To the ale gien obserations of all the i ales, copte argax 3 Naïe Bayes Classifier argax 4 7

8 8 5 Naïe Bayes Classifier argax, argax 6 Naïe Bayes Classifier argax, argax argax

9 9 7 Naïe Bayes Classifier argax argax, argax argax 8 Naïe Bayes Classifier argax argax argax, argax argax Becase of the strctre of the Bayes Net

10 Technical oint # The probabilities = = can soeties be really sall This can reslt in nerical instability since floating point nbers are not represented exactly on any copter architectre To get arond this, se the log of the last line in the preios slide i.e. argaxlog log 9 Technical oint #2 When estiating paraeters, what happens if yo don t hae any records that atch a certain cobination of featres? For exaple, in or training data, we didn t hae M=false, A=false, G=false, W=false This eans that = = in the forla below will be 0 and the entire expression will be 0. Een ore horrible things happen if yo had this expression in log space 20 0

11 Unifor Dirichlet riors Let N be the nber of ales that can take on. #records with and #records with N What happens when yo hae no records with =? N This eans that each ale of is eqally likely in the absence of data. If yo hae a lot of data, it doinates the /N ale. We call this trick a nifor Dirichlet prior. 2 Exaple Monday Assn Grades Win CD tre tre tre false tre false tre tre tre false tre false false false false false tre false false tre tre tre tre false tre false false tre false tre tre tre false tre false Copte MCD sing nifor Dirichlet priors 22

12 ractice Monday Assn Grades Win CD tre tre tre false tre false tre tre tre false tre false false false false false tre false false tre tre tre tre false tre false false tre false tre tre tre false tre false Copte W=treCD=tre sing nifor Dirichlet priors 23 rograing Assignent #3 o will classify text into two classes. There are two files:. Training data: trainingset.txt 2. Testing data: testset.txt 24 2

13 rograing Assignent #3 Two parts to this assignent:. re-processing step 2. Classification step 25. reprocessing Step Recall that naïe Bayes has the strctre shown to the right The nodes correspond to rando ariables, which are the featres or attribtes in the data What are the featres in the docents? Note: a docent in or assignent is a elp reiew to be classified as positie or negatie 26 3

14 The Vocablary The featres of the docents will be the presence/absence of words in the ocablary The ocablary is the list of words that are known to the classifier Ideally, the ocablary wold be all the words in the English langage For this assignent, yo will for the ocablary sing all the words in the training data 27 Bag of Words Sppose yo hae the following docents: Training Data This is an excellent laptop No, this is not sarcas! Class Label Class Class 0 Test Data Excellent Laptop = Class o will ignore pnctation for this assignent The ocablary will be: this, is, an, excellent, laptop, no, not, sarcas 28 4

15 Bag of Words Vocab: this, is, an, excellent, laptop, no, not, sarcas Keep this in alphabetical order to help with debgging Vocab: an, excellent, is, laptop, no, not, sarcas, this 29 Training data Next, conert yor training and test data into featres Training Data an excellent is laptop no not sarcas this Class Label Test Data an excellent is laptop no not sarcas this Class Label o will otpt the training data in featre for, with the featres alphabetized we will grade yo on this otpt. 30 5

16 2. Classification Step Training hase Class Label an excellent is laptop no not sarcas this or naïe Bayes classifier now looks soething like the aboe o still need to fill in the conditional probability tables in each node This is done in the training phase as described on slides 9 and 0 Reeber to se the nifor Dirichlet prior trick see slide Classification Step Testing hase Testing phase Load the featrized test data For each docent in the test data, its class label This reqires copting: Class label Words in docent 32 6

17 2. Classification Step Testing hase Sppose yo hae the following test instance: an excellent is laptop no not sarcas this Class Label to be ed Class = an = 0, excellent =, is = 0, laptop =, no = 0, not = 0, sarcas = 0, this = 0 = α Class = an = 0 Class = excellent = Class = is = 0 Class = laptop = Class = no=0class = not = 0 Class = sarcas = 0 Class = this = 0Class = Note: Use Word = Class if yo hae a for the word. Otherwise se Word = 0 Class Classification Step Testing hase an excellent is laptop no not sarcas this Class Label to be ed Then copte the following: Class = 0 an = 0, excellent =, is = 0, laptop =, no = 0, not = 0, sarcas = 0, this = 0 = α Class = 0 an = 0 Class = 0 excellent = Class = 0 is = 0 Class = 0 laptop = Class = 0 no=0class =0 not = 0 Class = 0 sarcas = 0 Class = 0 this = 0Class =

18 2. Classification Step Testing hase an excellent is laptop no not sarcas this Class Label to be ed If αclass = an = 0, excellent =, is = 0, laptop =, no = 0, not = 0, sarcas = 0, this = 0 > α Class = 0 an = 0, excellent =, is = 0, laptop =, no = 0, not = 0, sarcas = 0, this = 0 redict Class = otherwise Class = Classification Step Testing hase For each docent in the testing data set, its class label Copare the ed class label to the actal class label Otpt the accracy for each class: # correctly ed class labels total # of ion s 36 8

19 Reslts There are two sets of reslts we reqire:. Reslts #: Use trainingset.txt for the training phase Use trainingset.txt for the testing phase Report accracy 2. Reslts #2: Use trainingset.txt for the training phase Use testset.txt for the testing phase Report accracy 37 What o Shold Know How to learn the paraeters for a Naïe Bayes odel How to ake ions with a Naïe Bayes odel How to ipleent a Naïe Bayes Model 38 9