Naïve Bayes Introduction to Machine Learning. Matt Gormley Lecture 3 September 14, Readings: Mitchell Ch Murphy Ch.
|
|
- Jennifer Miles
- 5 years ago
- Views:
Transcription
1 School of Computer Science Introduction to Machine Learning aïve Bayes Readings: Mitchell Ch Murphy Ch. 3 Matt Gormley Lecture 3 September 14,
2 Homewor 1: due 9/26/16 Project Proposal: due 10/3/16 start early! Reminders 2
3 Outline Bacground: Maximum lielihood estimation (MLE) Maximum a posteriori (MAP) estimation Example: Exponential distribution Generative Models Model 0: ot- so- naïve Model aïve Bayes aïve Bayes Assumption Model 1: Bernoulli aïve Bayes Model 2: Multinomial aïve Bayes Model 3: Gaussian aïve Bayes Model 4: Multiclass aïve Bayes Smoothing Add- 1 Smoothing Add- λ Smoothing MAP Estimation (Beta Prior) 3
4 MLE vs. MAP Suppose we have data D = {x (i) } i=1 MLE = p( (i) ) Maximum Lielihood Estimate (MLE) i=1 4
5 Bacground: MLE Example: MLE of Exponential Distribution pdf of Exponential( ): f(x) = e x Suppose X i Exponential( ) for 1 i. Find MLE for data D = {x (i) } i=1 First write down log-lielihood of sample. Compute first derivative, set to zero, solve for. Compute second derivative and chec that it is concave down at MLE. 5
6 Bacground: MLE Example: MLE of Exponential Distribution First write down log-lielihood of sample. ( )= = = i=1 i=1 i=1 = ( ) f(x (i) ) (1) ( ( x (i) )) (2) ( )+ x (i) (3) i=1 x (i) (4) 6
7 Bacground: MLE Example: MLE of Exponential Distribution Compute first derivative, set to zero, solve for. d ( ) d = d d ( ) = i=1 MLE = i=1 x (i) (1) x (i) =0 (2) i=1 x(i) (3) 7
8 Bacground: MLE Example: MLE of Exponential Distribution pdf of Exponential( ): f(x) = e x Suppose X i Exponential( ) for 1 i. Find MLE for data D = {x (i) } i=1 First write down log-lielihood of sample. Compute first derivative, set to zero, solve for. Compute second derivative and chec that it is concave down at MLE. 8
9 MLE vs. MAP Suppose we have data D = {x (i) } i=1 MLE = MAP = i=1 i=1 p( (i) ) p( (i) )p( ) Maximum Lielihood Estimate (MLE) Maximum a posteriori (MAP) estimate Prior 9
10 Generative Models Specify a generative story for how the data was created (e.g. roll a weighted die) Given parameters (e.g. weights for each side) for the model, we can generate new data (e.g. roll the die again) Typical learning approach is MLE (e.g. find the most liely weights given the data) 10
11 Features Suppose we want to represent a document (M words) as a vector (vocabulary size V ) How should we do it? Option 1: Integer vector (word IDs) =[x 1,x 2,...,x M ] where x m {1,...,K} a word id. Option 2: Binary vector (word indicators) =[x 1,x 2,...,x K ] where x {0, 1} is a boolean. Option 3: Integer vector (word counts) =[x 1,x 2,...,x K ] where x Z + is a positive integer. 11
12 Today s Goal To define a generative model of s of two different classes (e.g. spam vs. not spam) 12
13 Spam ews The Economist The Onion 13
14 Model 0: ot- so- naïve Model? Generative Story: 1. Flip a weighted coin (Y) 2. If heads, roll the red many sided die to sample a document vector (X) from the Spam distribution 3. If tails, roll the blue many sided die to sample a document vector (X) from the ot- Spam distribution P (X 1,...,X K,Y)=P (X 1,...,X K Y )P (Y ) This model is computationally naïve! 14
15 Model 0: ot- so- naïve Model? Generative Story: 1. Flip a weighted coin (Y) 2. If heads, sample a document ID (X) from the Spam distribution 3. If tails, sample a document ID (X) from the ot- Spam distribution P (X, Y )=P (X Y )P (Y ) This model is computationally naïve! 15
16 Model 0: ot- so- naïve Model? Flip weighted coin If HEADS, roll red die y x 1 x 2 x 3 x K If TAILS, roll blue die Each side of the die is labeled with a document vector (e.g. [1,0,1,,1])
17 Outline Bacground: Maximum lielihood estimation (MLE) Maximum a posteriori (MAP) estimation Example: Exponential distribution Generative Models Model 0: ot- so- naïve Model aïve Bayes aïve Bayes Assumption Model 1: Bernoulli aïve Bayes Model 2: Multinomial aïve Bayes Model 3: Gaussian aïve Bayes Model 4: Multiclass aïve Bayes Smoothing Add- 1 Smoothing Add- λ Smoothing MAP Estimation (Beta Prior) 17
18 aïve Bayes Assumption Conditional independence of features: P (X 1,...,X K,Y)=P(X 1,...,X K Y )P (Y ) = K =1 P (X Y ) P (Y ) 18
19 C P(C) A C P(A C) B C P(B C) Estimating a joint from conditional probabilities P(A, B C) = P(A C)* P(B C) a, bc : P(A = a B = b C = c) = P(A = a C = c)* P(B = b C = c) A B C P(A,B,C)
20 C P(C) A C P(A C) B 0.5 C P(B C) D C P(D C) Estimating a joint from conditional probabilities A B D C P(A,B,D,C)
21 Assuming conditional independence, the conditional probabilities encode the same information as the joint table. They are very convenient for estimating P( X 1,,X n Y)=P( X 1 Y)* *P( X n Y) They are almost as good for computing P(Y X 1,..., X n ) = P(X..., X 1, n Y )P(Y ) P(X 1,..., X n ) x,y : P(Y = y X 1,..., X n = x) = P(X..., X 1, n = x Y)P(Y = y) P(X 1,..., X n = x)
22 Generic aïve Bayes Model Support: Depends on the choice of event model, P(X Y) Model: Product of prior and the event model P (,Y)=P (Y ) K =1 P (X Y ) Training: Find the class- conditional MLE parameters For P(Y), we find the MLE using all the data. For each P(X Y) we condition on the data with the corresponding class. Classification: Find the class that maximizes the posterior ŷ = y p(y ) 22
23 Generic aïve Bayes Model Classification: ŷ = y = y = y p(y ) p( y)p(y) p(x) p( y)p(y) (posterior) (by Bayes rule) 23
24 Model 1: Bernoulli aïve Bayes Support: Binary vectors of length K {0, 1} K Generative Story: Y Bernoulli( ) X Bernoulli(,Y ) {1,...,K} Model: p, (x,y)=p, (x 1,...,x K,y) = p (y) K =1 p (x y) =( ) y (1 ) (1 y) K =1 (,y ) x (1,y) (1 x ) 24
25 Model 0: ot- so- naïve Model? Flip weighted coin If HEADS, flip each red coin If TAILS, flip each blue coin y x 1 x 2 x 3 x K Each red coin corresponds to an x We can generate data in this fashion. Though in practice we never would since our data is given. Instead, this provides an explanation of how the data was generated (albeit a terrible one). 25
26 Model 1: Bernoulli aïve Bayes Support: Binary vectors of length K {0, 1} K Generative Story: Y Bernoulli( ) X Bernoulli(,Y ) {1,...,K} Model: p, (x,y) =( ) y (1 ) (1 y) K =1 Same as Generic aïve Bayes (,y ) x (1,y) (1 x ) Classification: Find the class that maximizes the posterior ŷ = y p(y ) 26
27 Generic aïve Bayes Model Recall Classification: ŷ = y = y = y p(y ) p( y)p(y) p(x) p( y)p(y) (posterior) (by Bayes rule) 27
28 Model 1: Bernoulli aïve Bayes Training: Find the class- conditional MLE parameters For P(Y), we find the MLE using all the data. For each P(X Y) we condition on the data with the corresponding class. =,0 =,1 = i=1 I(y(i) = 1) i=1 I(y(i) =0 x (i) = 1) i=1 I(y(i) = 0) i=1 I(y(i) =1 x (i) = 1) i=1 I(y(i) = 1) {1,...,K} 28
29 Model 1: Bernoulli aïve Bayes Training: Find the class- conditional MLE parameters For P(Y), we find the MLE using all the data. For each P(X Y) we condition on the data with the corresponding class. =,0 =,1 = i=1 I(y(i) = 1) i=1 I(y(i) =0 x (i) = 1) i=1 I(y(i) = 0) i=1 I(y(i) =1 x (i) = 1) i=1 I(y(i) = 1) Data: y x 1 x 2 x 3 x K {1,...,K}
30 Outline Bacground: Maximum lielihood estimation (MLE) Maximum a posteriori (MAP) estimation Example: Exponential distribution Generative Models Model 0: ot- so- naïve Model aïve Bayes aïve Bayes Assumption Model 1: Bernoulli aïve Bayes Model 2: Multinomial aïve Bayes Model 3: Gaussian aïve Bayes Model 4: Multiclass aïve Bayes Smoothing Add- 1 Smoothing Add- λ Smoothing MAP Estimation (Beta Prior) 30
31 Smoothing 1. Add- 1 Smoothing 2. Add- λ Smoothing 3. MAP Estimation (Beta Prior) 31
32 MLE What does maximizing lielihood accomplish? There is only a finite amount of probability mass (i.e. sum- to- one constraint) MLE tries to allocate as much probability mass as possible to the things we have observed at the expense of the things we have not observed 32
33 MLE For aïve Bayes, suppose we never observe the word serious in an Onion article. In this case, what is the MLE of p(x y)?,0 = i=1 I(y(i) =0 x (i) = 1) i=1 I(y(i) = 0) ow suppose we observe the word serious at test time. What is the posterior probability that the article was an Onion article? p(y x) = p(x y)p(y) p(x) 33
34 1. Add- 1 Smoothing The simplest setting for smoothing simply adds a single pseudo-observation to the data. This converts the true observations D into a new dataset D from we derive the MLEs. D = {( (i),y (i) )} i=1 (1) D = D {(, 0), (, 1), (, 0), (, 1)} (2) where is the vector of all zeros and is the vector of all ones. This has the effect of pretending that we observed each feature x with each class y. 34
35 1. Add- 1 Smoothing What if we write the MLEs in terms of the original dataset D? = i=1 I(y(i) = 1),0 = 1+ i=1 I(y(i) =0 x (i) = 1) 2+ i=1 I(y(i) = 0),1 = 1+ i=1 I(y(i) =1 x (i) = 1) 2+ i=1 I(y(i) = 1) {1,...,K} 35
36 2. Add- λ Smoothing For the Categorical Distribution Suppose we have a dataset obtained by repeatedly rolling a K-sided (weighted) die. Given data D = {x (i) } i=1 where x(i) {1,...,K}, we have the following MLE: = i=1 I(x(i) = ) With add- smoothing, we add pseudo-observations as before to obtain a smoothed estimate: = + i=1 I(x(i) = ) + 36
37 MLE vs. MAP Recall Suppose we have data D = {x (i) } i=1 MLE = MAP = i=1 i=1 p( (i) ) p( (i) )p( ) Maximum Lielihood Estimate (MLE) Maximum a posteriori (MAP) estimate Prior 37
38 3. MAP Estimation (Beta Prior) Generative Story: The parameters are drawn once for the entire dataset. for for i {1,...,K}: for y {0, 1}:,y Beta(, ) {1,...,}: y (i) Bernoulli( ) for x (i) {1,...,K}: Bernoulli(,y (i)) Training: Find the class- conditional MAP parameters = i=1 I(y(i) = 1),0 = ( 1) + i=1 I(y(i) =0 x (i) = 1) ( 1) + ( 1) + i=1 I(y(i) = 0),1 = ( 1) + i=1 I(y(i) =1 x (i) = 1) ( 1) + ( 1) + i=1 I(y(i) = 1) {1,...,K} 38
39 Outline Bacground: Maximum lielihood estimation (MLE) Maximum a posteriori (MAP) estimation Example: Exponential distribution Generative Models Model 0: ot- so- naïve Model aïve Bayes aïve Bayes Assumption Model 1: Bernoulli aïve Bayes Model 2: Multinomial aïve Bayes Model 3: Gaussian aïve Bayes Model 4: Multiclass aïve Bayes Smoothing Add- 1 Smoothing Add- λ Smoothing MAP Estimation (Beta Prior) 39
40 Model 2: Multinomial aïve Bayes Support: Option 1: Integer vector (word IDs) =[x 1,x 2,...,x M ] where x m {1,...,K} a word id. Generative Story: for i {1,...,}: y (i) Bernoulli( ) for j {1,...,M i }: x (i) j Multinomial( y (i), 1) Model: p, (x,y)=p (y) K =1 p (x y) =( ) y (1 ) (1 y) M i j=1 y,x j 40
41 Gaussian Discriminative Analysis l learning f: X Y, where l X is a vector of real-valued features, X n = < X n1, X n m > l Y is an indicator vector l What does that imply about the form of P(Y X)? l The joint probability of a datum and its label is: l Given a datum x n, we predict its label using the conditional probability of the label given the datum: Eric CMU, Y n X n { } ) - ( ) - ( exp - ) (2 1 ), 1, ( 1) ( ), 1, ( / 1 n T n n n n n n y p y p y p µ µ π π µ σ µ!! x x x x Σ Σ = Σ = = = = { } { } Σ Σ Σ Σ = Σ = ' ' / ' / ) - ( ) - ( - exp ) (2 1 ) - ( ) - ( - exp ) (2 1 ),, 1 ( n T n n T n n y n p µ µ π π µ µ π π µ!!!! x x x x x
42 Model 3: Gaussian aïve Bayes Support: Model: Product of prior and the event model R K p(x,y)=p(x 1,...,x K,y) = p(y) K =1 p(x y) Gaussian aive Bayes assumes that p(x y) is given by a ormal distribution. 42
43 Gaussian aïve Bayes Classifier l When X is multivariate-gaussian vector: l The joint probability of a datum and it label is: l The naïve Bayes simplification l More generally: l Where p(..) is an arbitrary conditional (discrete or continuous) 1-D density Eric CMU, { } ) - ( ) - ( exp - ) (2 1 ), 1, ( 1) ( ), 1, ( / 1 n T n n n n n n y p y p y p µ µ π π µ µ!!!! x x x x Σ Σ = Σ = = = Σ = Y n X n = = = = = j j j j n j j j j n j n n n n x y x p y p y p exp 2 1 ), 1, ( 1) ( ), 1, ( σ µ πσ π σ µ σ µ x = = m j n j n n n n y x p y p y p 1 ), ( ) ( ),, ( η π π η x Y n X n,1 X n,2 X n,m
44 VISUALIZIG AÏVE BAYES Slides in this section from William Cohen (10-601B, Spring 2016) 44
45
46 Fisher Iris Dataset Fisher (1936) used 150 measurements of flowers from 3 different species: Iris setosa (0), Iris virginica (1), Iris versicolor (2) collected by Anderson (1936) Species Sepal Length Sepal Width Petal Length Petal Width Full dataset: 46
47 %% Import the IRIS data load fisheriris; X = meas; pos = strcmp(species,'setosa'); Y = 2*pos - 1; %% Visualize the data imagesc([x,y]); title('iris data');
48 %% Visualize by scatter plotting the the first two dimensions figure; scatter(x(y<0,1),x(y<0,2),'r*'); hold on; scatter(x(y>0,1),x(y>0,2),'bo'); title('iris data');
49 %% Compute the mean and SD of each class PosMean = mean(x(y>0,:)); PosSD = std(x(y>0,:)); egmean = mean(x(y<0,:)); egsd = std(x(y<0,:)); %% Compute the B probabilities for each class for each grid element [G1,G2]=meshgrid(3:0.1:8, 2:0.1:5); Z1 = gaussmf(g1,[possd(1),posmean(1)]); Z2 = gaussmf(g2,[possd(2),posmean(2)]); Z = Z1.* Z2; V1 = gaussmf(g1,[egsd(1),egmean(1)]); V2 = gaussmf(g2,[egsd(2),egmean(2)]); V = V1.* V2;
50 % Add them to the scatter plot figure; scatter(x(y<0,1),x(y<0,2),'r*'); hold on; scatter(x(y>0,1),x(y>0,2),'bo'); contour(g1,g2,z); contour(g1,g2,v);
51 %% ow plot the difference of the probabilities figure; scatter(x(y<0,1),x(y<0,2),'r*'); hold on; scatter(x(y>0,1),x(y>0,2),'bo'); contour(g1,g2,z- V); mesh(g1,g2,z- V); alpha(0.4)
52 AÏVE BAYES IS LIEAR Slides in this section from William Cohen (10-601B, Spring 2016)
53 Question: what does the boundary between positive and negative loo lie for aïve Bayes?
54
55 argmax y P(X i = x i Y = y ) P(Y = y ) i = argmax y log P(X i = x i Y = y ) + log P(Y = y ) i = argmax y {+1, 1} log P(x i y ) + log P(y ) # & = sign% log P(x i y +1 ) log P(x i y 1 ) + log P(y +1 ) log P(y 1 )( $ ' i i i two classes only # = sign log P(x i y +1 ) P(x i y 1 ) + log P(y +1) & % ( $ P(y 1 )' i rearrange terms
56 argmax y P(X i = x i Y = y ) P(Y = y ) i # = sign log P(x i y +1 ) P(x i y 1 ) + log P(y +1) & % ( $ P(y 1 )' i if x i = 1 or 0. " u i = log P(x =1 y ) i +1 P(x i =1 y 1 ) log P(x = 0 y ) % i +1 $ ' # P(x i = 0 y 1 )& " " = sign x i log P(x =1 y ) i +1 P(x i =1 y 1 ) log P(x = 0 y ) % " i +1 $ ' + log P(x = 0 y ) % i +1 $ ' + log P(y ) % +1 $ # i # P(x i = 0 y 1 )& i # P(x i = 0 y 1 )& P(y 1 ) ' & " % = sign$ x i u i + u 0 ' # i & " % = sign$ x i u i + x 0 u 0 ' # i & x 0 =1 for every x (bias term) = sign x u ( ) " u 0 = log P(x i = 0 y +1 )% $ ' + log P(y +1) # P(x i = 0 y 1 )& P(y 1 ) i
57
58
59 Why don t we drop the generative model and try to learn this hyperplane directly?
60 Model 4: Multiclass aïve Bayes Model: The only change is that we permit y to range over C classes. p(x,y)=p(x 1,...,x K,y) = p(y) K =1 p(x y) ow, y Multinomial(, 1) and we have a separate conditional distribution p(x y) for each of the C classes. 60
61 aïve Bayes Assumption Recall Conditional independence of features: P (X 1,...,X K,Y)=P(X 1,...,X K Y )P (Y ) = K =1 P (X Y ) P (Y ) 61
62 =P (X 1,...,X K Y )P (Y ) = K =1 P (X Y ) P (Y ) What s wrong with the aïve Bayes Assumption? The features might not be independent!! Example 1: If a document contains the word Donald, it s extremely liely to contain the word Trump These are not independent! Example 2: If the petal width is very high, the petal length is also liely to be very high 62
63 Summary 1. aïve Bayes provides a framewor for generative modeling 2. Choose an event model appropriate to the data 3. Train by MLE or MAP 4. Classify by maximizing the posterior 63
MLE/MAP + Naïve Bayes
10-601 Introduction to Machine Learning Machine Learning Department School of Computer Science Carnegie Mellon University MLE/MAP + Naïve Bayes MLE / MAP Readings: Estimating Probabilities (Mitchell, 2016)
More informationNaïve Bayes Introduction to Machine Learning. Matt Gormley Lecture 18 Oct. 31, 2018
10-601 Introduction to Machine Learning Machine Learning Department School of Computer Science Carnegie Mellon University Naïve Bayes Matt Gormley Lecture 18 Oct. 31, 2018 1 Reminders Homework 6: PAC Learning
More informationMLE/MAP + Naïve Bayes
10-601 Introduction to Machine Learning Machine Learning Department School of Computer Science Carnegie Mellon University MLE/MAP + Naïve Bayes Matt Gormley Lecture 19 March 20, 2018 1 Midterm Exam Reminders
More informationProbabilistic modeling. The slides are closely adapted from Subhransu Maji s slides
Probabilistic modeling The slides are closely adapted from Subhransu Maji s slides Overview So far the models and algorithms you have learned about are relatively disconnected Probabilistic modeling framework
More informationNaïve Bayes classification
Naïve Bayes classification 1 Probability theory Random variable: a variable whose possible values are numerical outcomes of a random phenomenon. Examples: A person s height, the outcome of a coin toss
More informationBayesian Methods: Naïve Bayes
Bayesian Methods: aïve Bayes icholas Ruozzi University of Texas at Dallas based on the slides of Vibhav Gogate Last Time Parameter learning Learning the parameter of a simple coin flipping model Prior
More informationThe Naïve Bayes Classifier. Machine Learning Fall 2017
The Naïve Bayes Classifier Machine Learning Fall 2017 1 Today s lecture The naïve Bayes Classifier Learning the naïve Bayes Classifier Practical concerns 2 Today s lecture The naïve Bayes Classifier Learning
More informationNaïve Bayes classification. p ij 11/15/16. Probability theory. Probability theory. Probability theory. X P (X = x i )=1 i. Marginal Probability
Probability theory Naïve Bayes classification Random variable: a variable whose possible values are numerical outcomes of a random phenomenon. s: A person s height, the outcome of a coin toss Distinguish
More informationIntroduction to Bayesian Learning. Machine Learning Fall 2018
Introduction to Bayesian Learning Machine Learning Fall 2018 1 What we have seen so far What does it mean to learn? Mistake-driven learning Learning by counting (and bounding) number of mistakes PAC learnability
More informationBayesian Learning. HT2015: SC4 Statistical Data Mining and Machine Learning. Maximum Likelihood Principle. The Bayesian Learning Framework
HT5: SC4 Statistical Data Mining and Machine Learning Dino Sejdinovic Department of Statistics Oxford http://www.stats.ox.ac.uk/~sejdinov/sdmml.html Maximum Likelihood Principle A generative model for
More informationMachine Learning
Machine Learning 10-601 Tom M. Mitchell Machine Learning Department Carnegie Mellon University January 26, 2015 Today: Bayes Classifiers Conditional Independence Naïve Bayes Readings: Mitchell: Naïve Bayes
More informationLearning Bayesian network : Given structure and completely observed data
Learning Bayesian network : Given structure and completely observed data Probabilistic Graphical Models Sharif University of Technology Spring 2017 Soleymani Learning problem Target: true distribution
More informationCheng Soon Ong & Christian Walder. Canberra February June 2018
Cheng Soon Ong & Christian Walder Research Group and College of Engineering and Computer Science Canberra February June 2018 (Many figures from C. M. Bishop, "Pattern Recognition and ") 1of 305 Part VII
More informationMachine Learning
Machine Learning 10-601 Tom M. Mitchell Machine Learning Department Carnegie Mellon University September 22, 2011 Today: MLE and MAP Bayes Classifiers Naïve Bayes Readings: Mitchell: Naïve Bayes and Logistic
More informationCS 361: Probability & Statistics
October 17, 2017 CS 361: Probability & Statistics Inference Maximum likelihood: drawbacks A couple of things might trip up max likelihood estimation: 1) Finding the maximum of some functions can be quite
More informationHidden Markov Models
10-601 Introduction to Machine Learning Machine Learning Department School of Computer Science Carnegie Mellon University Hidden Markov Models Matt Gormley Lecture 22 April 2, 2018 1 Reminders Homework
More informationClassification Methods II: Linear and Quadratic Discrimminant Analysis
Classification Methods II: Linear and Quadratic Discrimminant Analysis Rebecca C. Steorts, Duke University STA 325, Chapter 4 ISL Agenda Linear Discrimminant Analysis (LDA) Classification Recall that linear
More informationLogistic Regression Introduction to Machine Learning. Matt Gormley Lecture 9 Sep. 26, 2018
10-601 Introduction to Machine Learning Machine Learning Department School of Computer Science Carnegie Mellon University Logistic Regression Matt Gormley Lecture 9 Sep. 26, 2018 1 Reminders Homework 3:
More informationNaïve Bayes. Jia-Bin Huang. Virginia Tech Spring 2019 ECE-5424G / CS-5824
Naïve Bayes Jia-Bin Huang ECE-5424G / CS-5824 Virginia Tech Spring 2019 Administrative HW 1 out today. Please start early! Office hours Chen: Wed 4pm-5pm Shih-Yang: Fri 3pm-4pm Location: Whittemore 266
More informationToday. Statistical Learning. Coin Flip. Coin Flip. Experiment 1: Heads. Experiment 1: Heads. Which coin will I use? Which coin will I use?
Today Statistical Learning Parameter Estimation: Maximum Likelihood (ML) Maximum A Posteriori (MAP) Bayesian Continuous case Learning Parameters for a Bayesian Network Naive Bayes Maximum Likelihood estimates
More informationGaussian discriminant analysis Naive Bayes
DM825 Introduction to Machine Learning Lecture 7 Gaussian discriminant analysis Marco Chiarandini Department of Mathematics & Computer Science University of Southern Denmark Outline 1. is 2. Multi-variate
More informationMachine Learning. Regression-Based Classification & Gaussian Discriminant Analysis. Manfred Huber
Machine Learning Regression-Based Classification & Gaussian Discriminant Analysis Manfred Huber 2015 1 Logistic Regression Linear regression provides a nice representation and an efficient solution to
More informationThe Bayes classifier
The Bayes classifier Consider where is a random vector in is a random variable (depending on ) Let be a classifier with probability of error/risk given by The Bayes classifier (denoted ) is the optimal
More informationLogistic Regression Introduction to Machine Learning. Matt Gormley Lecture 8 Feb. 12, 2018
10-601 Introduction to Machine Learning Machine Learning Department School of Computer Science Carnegie Mellon University Logistic Regression Matt Gormley Lecture 8 Feb. 12, 2018 1 10-601 Introduction
More informationIntroduction to Machine Learning
Outline Introduction to Machine Learning Bayesian Classification Varun Chandola March 8, 017 1. {circular,large,light,smooth,thick}, malignant. {circular,large,light,irregular,thick}, malignant 3. {oval,large,dark,smooth,thin},
More informationCPSC 340: Machine Learning and Data Mining
CPSC 340: Machine Learning and Data Mining MLE and MAP Original version of these slides by Mark Schmidt, with modifications by Mike Gelbart. 1 Admin Assignment 4: Due tonight. Assignment 5: Will be released
More informationMachine Learning. Theory of Classification and Nonparametric Classifier. Lecture 2, January 16, What is theoretically the best classifier
Machine Learning 10-701/15 701/15-781, 781, Spring 2008 Theory of Classification and Nonparametric Classifier Eric Xing Lecture 2, January 16, 2006 Reading: Chap. 2,5 CB and handouts Outline What is theoretically
More informationIntroduction to Probability and Statistics (Continued)
Introduction to Probability and Statistics (Continued) Prof. icholas Zabaras Center for Informatics and Computational Science https://cics.nd.edu/ University of otre Dame otre Dame, Indiana, USA Email:
More informationLogistic Regression Review Fall 2012 Recitation. September 25, 2012 TA: Selen Uguroglu
Logistic Regression Review 10-601 Fall 2012 Recitation September 25, 2012 TA: Selen Uguroglu!1 Outline Decision Theory Logistic regression Goal Loss function Inference Gradient Descent!2 Training Data
More informationRepresentation. Stefano Ermon, Aditya Grover. Stanford University. Lecture 2
Representation Stefano Ermon, Aditya Grover Stanford University Lecture 2 Stefano Ermon, Aditya Grover (AI Lab) Deep Generative Models Lecture 2 1 / 32 Learning a generative model We are given a training
More informationLogistic Regression. William Cohen
Logistic Regression William Cohen 1 Outline Quick review classi5ication, naïve Bayes, perceptrons new result for naïve Bayes Learning as optimization Logistic regression via gradient ascent Over5itting
More informationEstimating Parameters
Machine Learning 10-601 Tom M. Mitchell Machine Learning Department Carnegie Mellon University September 13, 2012 Today: Bayes Classifiers Naïve Bayes Gaussian Naïve Bayes Readings: Mitchell: Naïve Bayes
More informationMachine Learning, Fall 2012 Homework 2
0-60 Machine Learning, Fall 202 Homework 2 Instructors: Tom Mitchell, Ziv Bar-Joseph TA in charge: Selen Uguroglu email: sugurogl@cs.cmu.edu SOLUTIONS Naive Bayes, 20 points Problem. Basic concepts, 0
More informationISyE 6416: Computational Statistics Spring Lecture 5: Discriminant analysis and classification
ISyE 6416: Computational Statistics Spring 2017 Lecture 5: Discriminant analysis and classification Prof. Yao Xie H. Milton Stewart School of Industrial and Systems Engineering Georgia Institute of Technology
More informationIntroduction to Machine Learning Spring 2018 Note 18
CS 189 Introduction to Machine Learning Spring 2018 Note 18 1 Gaussian Discriminant Analysis Recall the idea of generative models: we classify an arbitrary datapoint x with the class label that maximizes
More informationMachine Learning for Signal Processing Bayes Classification and Regression
Machine Learning for Signal Processing Bayes Classification and Regression Instructor: Bhiksha Raj 11755/18797 1 Recap: KNN A very effective and simple way of performing classification Simple model: For
More informationBayesian Learning (II)
Universität Potsdam Institut für Informatik Lehrstuhl Maschinelles Lernen Bayesian Learning (II) Niels Landwehr Overview Probabilities, expected values, variance Basic concepts of Bayesian learning MAP
More informationClassification CE-717: Machine Learning Sharif University of Technology. M. Soleymani Fall 2012
Classification CE-717: Machine Learning Sharif University of Technology M. Soleymani Fall 2012 Topics Discriminant functions Logistic regression Perceptron Generative models Generative vs. discriminative
More informationChapter 3: Maximum-Likelihood & Bayesian Parameter Estimation (part 1)
HW 1 due today Parameter Estimation Biometrics CSE 190 Lecture 7 Today s lecture was on the blackboard. These slides are an alternative presentation of the material. CSE190, Winter10 CSE190, Winter10 Chapter
More informationGenerative Learning algorithms
CS9 Lecture notes Andrew Ng Part IV Generative Learning algorithms So far, we ve mainly been talking about learning algorithms that model p(y x; θ), the conditional distribution of y given x. For instance,
More informationMachine Learning for Signal Processing Bayes Classification
Machine Learning for Signal Processing Bayes Classification Class 16. 24 Oct 2017 Instructor: Bhiksha Raj - Abelino Jimenez 11755/18797 1 Recap: KNN A very effective and simple way of performing classification
More informationCPSC 340: Machine Learning and Data Mining. MLE and MAP Fall 2017
CPSC 340: Machine Learning and Data Mining MLE and MAP Fall 2017 Assignment 3: Admin 1 late day to hand in tonight, 2 late days for Wednesday. Assignment 4: Due Friday of next week. Last Time: Multi-Class
More informationProbability Theory for Machine Learning. Chris Cremer September 2015
Probability Theory for Machine Learning Chris Cremer September 2015 Outline Motivation Probability Definitions and Rules Probability Distributions MLE for Gaussian Parameter Estimation MLE and Least Squares
More informationBayesian Models in Machine Learning
Bayesian Models in Machine Learning Lukáš Burget Escuela de Ciencias Informáticas 2017 Buenos Aires, July 24-29 2017 Frequentist vs. Bayesian Frequentist point of view: Probability is the frequency of
More informationCS4705. Probability Review and Naïve Bayes. Slides from Dragomir Radev
CS4705 Probability Review and Naïve Bayes Slides from Dragomir Radev Classification using a Generative Approach Previously on NLP discriminative models P C D here is a line with all the social media posts
More informationMachine Learning
Machine Learning 10-701 Tom M. Mitchell Machine Learning Department Carnegie Mellon University February 1, 2011 Today: Generative discriminative classifiers Linear regression Decomposition of error into
More informationLEC 4: Discriminant Analysis for Classification
LEC 4: Discriminant Analysis for Classification Dr. Guangliang Chen February 25, 2016 Outline Last time: FDA (dimensionality reduction) Today: QDA/LDA (classification) Naive Bayes classifiers Matlab/Python
More informationIntroduction: MLE, MAP, Bayesian reasoning (28/8/13)
STA561: Probabilistic machine learning Introduction: MLE, MAP, Bayesian reasoning (28/8/13) Lecturer: Barbara Engelhardt Scribes: K. Ulrich, J. Subramanian, N. Raval, J. O Hollaren 1 Classifiers In this
More informationHidden Markov Models
10-601 Introduction to Machine Learning Machine Learning Department School of Computer Science Carnegie Mellon University Hidden Markov Models Matt Gormley Lecture 19 Nov. 5, 2018 1 Reminders Homework
More informationCSC321 Lecture 18: Learning Probabilistic Models
CSC321 Lecture 18: Learning Probabilistic Models Roger Grosse Roger Grosse CSC321 Lecture 18: Learning Probabilistic Models 1 / 25 Overview So far in this course: mainly supervised learning Language modeling
More informationSome Probability and Statistics
Some Probability and Statistics David M. Blei COS424 Princeton University February 13, 2012 Card problem There are three cards Red/Red Red/Black Black/Black I go through the following process. Close my
More informationLecture 4. Generative Models for Discrete Data - Part 3. Luigi Freda. ALCOR Lab DIAG University of Rome La Sapienza.
Lecture 4 Generative Models for Discrete Data - Part 3 Luigi Freda ALCOR Lab DIAG University of Rome La Sapienza October 6, 2017 Luigi Freda ( La Sapienza University) Lecture 4 October 6, 2017 1 / 46 Outline
More informationBayesian Classifiers, Conditional Independence and Naïve Bayes. Required reading: Naïve Bayes and Logistic Regression (available on class website)
Bayesian Classifiers, Conditional Independence and Naïve Bayes Required reading: Naïve Bayes and Logistic Regression (available on class website) Machine Learning 10-701 Tom M. Mitchell Machine Learning
More informationCS 446 Machine Learning Fall 2016 Nov 01, Bayesian Learning
CS 446 Machine Learning Fall 206 Nov 0, 206 Bayesian Learning Professor: Dan Roth Scribe: Ben Zhou, C. Cervantes Overview Bayesian Learning Naive Bayes Logistic Regression Bayesian Learning So far, we
More informationMachine Learning Tom M. Mitchell Machine Learning Department Carnegie Mellon University. September 20, 2012
Machine Learning 10-601 Tom M. Mitchell Machine Learning Department Carnegie Mellon University September 20, 2012 Today: Logistic regression Generative/Discriminative classifiers Readings: (see class website)
More informationQualifier: CS 6375 Machine Learning Spring 2015
Qualifier: CS 6375 Machine Learning Spring 2015 The exam is closed book. You are allowed to use two double-sided cheat sheets and a calculator. If you run out of room for an answer, use an additional sheet
More informationProbabilistic classification CE-717: Machine Learning Sharif University of Technology. M. Soleymani Fall 2016
Probabilistic classification CE-717: Machine Learning Sharif University of Technology M. Soleymani Fall 2016 Topics Probabilistic approach Bayes decision theory Generative models Gaussian Bayes classifier
More informationMachine Learning
Machine Learning 10-601 Tom M. Mitchell Machine Learning Department Carnegie Mellon University August 30, 2017 Today: Decision trees Overfitting The Big Picture Coming soon Probabilistic learning MLE,
More informationIntroduc)on to Bayesian methods (con)nued) - Lecture 16
Introduc)on to Bayesian methods (con)nued) - Lecture 16 David Sontag New York University Slides adapted from Luke Zettlemoyer, Carlos Guestrin, Dan Klein, and Vibhav Gogate Outline of lectures Review of
More informationCHAPTER 2 Estimating Probabilities
CHAPTER 2 Estimating Probabilities Machine Learning Copyright c 2017. Tom M. Mitchell. All rights reserved. *DRAFT OF September 16, 2017* *PLEASE DO NOT DISTRIBUTE WITHOUT AUTHOR S PERMISSION* This is
More informationKEY CONCEPTS IN PROBABILITY: SMOOTHING, MLE, AND MAP
KEY CONCEPTS IN PROBABILITY: SMOOTHING, MLE, AND MAP Outline MAPs and MLEs catchup from last week Joint Distributions a new learner Naïve Bayes another new learner Administrivia Homeworks: Due tomorrow
More informationCS540 ANSWER SHEET
CS540 ANSWER SHEET Name Email 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 1 2 Final Examination CS540-1: Introduction to Artificial Intelligence Fall 2016 20 questions, 5 points
More informationPATTERN RECOGNITION AND MACHINE LEARNING CHAPTER 2: PROBABILITY DISTRIBUTIONS
PATTERN RECOGNITION AND MACHINE LEARNING CHAPTER 2: PROBABILITY DISTRIBUTIONS Parametric Distributions Basic building blocks: Need to determine given Representation: or? Recall Curve Fitting Binary Variables
More informationGenerative Learning. INFO-4604, Applied Machine Learning University of Colorado Boulder. November 29, 2018 Prof. Michael Paul
Generative Learning INFO-4604, Applied Machine Learning University of Colorado Boulder November 29, 2018 Prof. Michael Paul Generative vs Discriminative The classification algorithms we have seen so far
More informationMachine Learning. Classification. Bayes Classifier. Representing data: Choosing hypothesis class. Learning: h:x a Y. Eric Xing
Machine Learning 10-701/15 701/15-781, 781, Spring 2008 Naïve Bayes Classifier Eric Xing Lecture 3, January 23, 2006 Reading: Chap. 4 CB and handouts Classification Representing data: Choosing hypothesis
More informationLecture : Probabilistic Machine Learning
Lecture : Probabilistic Machine Learning Riashat Islam Reasoning and Learning Lab McGill University September 11, 2018 ML : Many Methods with Many Links Modelling Views of Machine Learning Machine Learning
More informationMachine Learning and Data Mining. Bayes Classifiers. Prof. Alexander Ihler
+ Machine Learning and Data Mining Bayes Classifiers Prof. Alexander Ihler A basic classifier Training data D={x (i),y (i) }, Classifier f(x ; D) Discrete feature vector x f(x ; D) is a con@ngency table
More informationMachine Learning
Machine Learning 10-601 Tom M. Mitchell Machine Learning Department Carnegie Mellon University February 4, 2015 Today: Generative discriminative classifiers Linear regression Decomposition of error into
More informationGenerative Clustering, Topic Modeling, & Bayesian Inference
Generative Clustering, Topic Modeling, & Bayesian Inference INFO-4604, Applied Machine Learning University of Colorado Boulder December 12-14, 2017 Prof. Michael Paul Unsupervised Naïve Bayes Last week
More informationCOS 424: Interacting with Data. Lecturer: Dave Blei Lecture #11 Scribe: Andrew Ferguson March 13, 2007
COS 424: Interacting with ata Lecturer: ave Blei Lecture #11 Scribe: Andrew Ferguson March 13, 2007 1 Graphical Models Wrap-up We began the lecture with some final words on graphical models. Choosing a
More informationCOMP 551 Applied Machine Learning Lecture 19: Bayesian Inference
COMP 551 Applied Machine Learning Lecture 19: Bayesian Inference Associate Instructor: (herke.vanhoof@mcgill.ca) Class web page: www.cs.mcgill.ca/~jpineau/comp551 Unless otherwise noted, all material posted
More informationNaïve Bayes. Vibhav Gogate The University of Texas at Dallas
Naïve Bayes Vibhav Gogate The University of Texas at Dallas Supervised Learning of Classifiers Find f Given: Training set {(x i, y i ) i = 1 n} Find: A good approximation to f : X Y Examples: what are
More informationCOMP90051 Statistical Machine Learning
COMP90051 Statistical Machine Learning Semester 2, 2017 Lecturer: Trevor Cohn 2. Statistical Schools Adapted from slides by Ben Rubinstein Statistical Schools of Thought Remainder of lecture is to provide
More informationOutline. Supervised Learning. Hong Chang. Institute of Computing Technology, Chinese Academy of Sciences. Machine Learning Methods (Fall 2012)
Outline Hong Chang Institute of Computing Technology, Chinese Academy of Sciences Machine Learning Methods (Fall 2012) Outline Outline I 1 Linear Models for Regression Linear Regression Probabilistic Interpretation
More informationIntroduction to Machine Learning
Introduction to Machine Learning Generative Models Varun Chandola Computer Science & Engineering State University of New York at Buffalo Buffalo, NY, USA chandola@buffalo.edu Chandola@UB CSE 474/574 1
More informationFundamentals. CS 281A: Statistical Learning Theory. Yangqing Jia. August, Based on tutorial slides by Lester Mackey and Ariel Kleiner
Fundamentals CS 281A: Statistical Learning Theory Yangqing Jia Based on tutorial slides by Lester Mackey and Ariel Kleiner August, 2011 Outline 1 Probability 2 Statistics 3 Linear Algebra 4 Optimization
More informationMachine Learning Gaussian Naïve Bayes Big Picture
Machine Learning 10-701 Tom M. Mitchell Machine Learning Department Carnegie Mellon University January 27, 2011 Today: Naïve Bayes Big Picture Logistic regression Gradient ascent Generative discriminative
More informationLinear Classification
Linear Classification Lili MOU moull12@sei.pku.edu.cn http://sei.pku.edu.cn/ moull12 23 April 2015 Outline Introduction Discriminant Functions Probabilistic Generative Models Probabilistic Discriminative
More informationLearning with Probabilities
Learning with Probabilities CS194-10 Fall 2011 Lecture 15 CS194-10 Fall 2011 Lecture 15 1 Outline Bayesian learning eliminates arbitrary loss functions and regularizers facilitates incorporation of prior
More informationUniversity of Cambridge Engineering Part IIB Module 4F10: Statistical Pattern Processing Handout 2: Multivariate Gaussians
Engineering Part IIB: Module F Statistical Pattern Processing University of Cambridge Engineering Part IIB Module F: Statistical Pattern Processing Handout : Multivariate Gaussians. Generative Model Decision
More informationMachine Learning
Machine Learning 10-701 Tom M. Mitchell Machine Learning Department Carnegie Mellon University January 13, 2011 Today: The Big Picture Overfitting Review: probability Readings: Decision trees, overfiting
More informationMachine Learning - MT & 5. Basis Expansion, Regularization, Validation
Machine Learning - MT 2016 4 & 5. Basis Expansion, Regularization, Validation Varun Kanade University of Oxford October 19 & 24, 2016 Outline Basis function expansion to capture non-linear relationships
More informationLogistic Regression. Jia-Bin Huang. Virginia Tech Spring 2019 ECE-5424G / CS-5824
Logistic Regression Jia-Bin Huang ECE-5424G / CS-5824 Virginia Tech Spring 2019 Administrative Please start HW 1 early! Questions are welcome! Two principles for estimating parameters Maximum Likelihood
More informationLogistic Regression. Machine Learning Fall 2018
Logistic Regression Machine Learning Fall 2018 1 Where are e? We have seen the folloing ideas Linear models Learning as loss minimization Bayesian learning criteria (MAP and MLE estimation) The Naïve Bayes
More informationContents Lecture 4. Lecture 4 Linear Discriminant Analysis. Summary of Lecture 3 (II/II) Summary of Lecture 3 (I/II)
Contents Lecture Lecture Linear Discriminant Analysis Fredrik Lindsten Division of Systems and Control Department of Information Technology Uppsala University Email: fredriklindsten@ituuse Summary of lecture
More informationGenerative classifiers: The Gaussian classifier. Ata Kaban School of Computer Science University of Birmingham
Generative classifiers: The Gaussian classifier Ata Kaban School of Computer Science University of Birmingham Outline We have already seen how Bayes rule can be turned into a classifier In all our examples
More informationProbabilistic Graphical Models
Probabilistic Graphical Models David Sontag New York University Lecture 4, February 16, 2012 David Sontag (NYU) Graphical Models Lecture 4, February 16, 2012 1 / 27 Undirected graphical models Reminder
More informationECE 5984: Introduction to Machine Learning
ECE 5984: Introduction to Machine Learning Topics: Classification: Logistic Regression NB & LR connections Readings: Barber 17.4 Dhruv Batra Virginia Tech Administrativia HW2 Due: Friday 3/6, 3/15, 11:55pm
More informationGaussian and Linear Discriminant Analysis; Multiclass Classification
Gaussian and Linear Discriminant Analysis; Multiclass Classification Professor Ameet Talwalkar Slide Credit: Professor Fei Sha Professor Ameet Talwalkar CS260 Machine Learning Algorithms October 13, 2015
More informationIntroduction to Machine Learning
Introduction to Machine Learning Bayesian Classification Varun Chandola Computer Science & Engineering State University of New York at Buffalo Buffalo, NY, USA chandola@buffalo.edu Chandola@UB CSE 474/574
More informationClassification objectives COMS 4771
Classification objectives COMS 4771 1. Recap: binary classification Scoring functions Consider binary classification problems with Y = { 1, +1}. 1 / 22 Scoring functions Consider binary classification
More informationMachine Learning
Machine Learning 10-601 Tom M. Mitchell Machine Learning Department Carnegie Mellon University February 2, 2015 Today: Logistic regression Generative/Discriminative classifiers Readings: (see class website)
More informationBowl Maximum Entropy #4 By Ejay Weiss. Maxent Models: Maximum Entropy Foundations. By Yanju Chen. A Basic Comprehension with Derivations
Bowl Maximum Entropy #4 By Ejay Weiss Maxent Models: Maximum Entropy Foundations By Yanju Chen A Basic Comprehension with Derivations Outlines Generative vs. Discriminative Feature-Based Models Softmax
More informationMachine Learning - MT Classification: Generative Models
Machine Learning - MT 2016 7. Classification: Generative Models Varun Kanade University of Oxford October 31, 2016 Announcements Practical 1 Submission Try to get signed off during session itself Otherwise,
More informationStatistical Data Mining and Machine Learning Hilary Term 2016
Statistical Data Mining and Machine Learning Hilary Term 2016 Dino Sejdinovic Department of Statistics Oxford Slides and other materials available at: http://www.stats.ox.ac.uk/~sejdinov/sdmml Naïve Bayes
More informationIntroduction to Machine Learning
1, DATA11002 Introduction to Machine Learning Lecturer: Antti Ukkonen TAs: Saska Dönges and Janne Leppä-aho Department of Computer Science University of Helsinki (based in part on material by Patrik Hoyer,
More informationIntroduction to machine learning and pattern recognition Lecture 2 Coryn Bailer-Jones
Introduction to machine learning and pattern recognition Lecture 2 Coryn Bailer-Jones http://www.mpia.de/homes/calj/mlpr_mpia2008.html 1 1 Last week... supervised and unsupervised methods need adaptive
More informationAarti Singh. Lecture 2, January 13, Reading: Bishop: Chap 1,2. Slides courtesy: Eric Xing, Andrew Moore, Tom Mitchell
Machine Learning 0-70/5 70/5-78, 78, Spring 00 Probability 0 Aarti Singh Lecture, January 3, 00 f(x) µ x Reading: Bishop: Chap, Slides courtesy: Eric Xing, Andrew Moore, Tom Mitchell Announcements Homework
More informationNon-parametric Methods
Non-parametric Methods Machine Learning Alireza Ghane Non-Parametric Methods Alireza Ghane / Torsten Möller 1 Outline Machine Learning: What, Why, and How? Curve Fitting: (e.g.) Regression and Model Selection
More informationSome Probability and Statistics
Some Probability and Statistics David M. Blei COS424 Princeton University February 12, 2007 D. Blei ProbStat 01 1 / 42 Who wants to scribe? D. Blei ProbStat 01 2 / 42 Random variable Probability is about
More information