Probabilistic classification CE-717: Machine Learning Sharif University of Technology. M. Soleymani Fall 2016
|
|
- Alaina Newman
- 5 years ago
- Views:
Transcription
1 Probabilistic classification CE-717: Machine Learning Sharif University of Technology M. Soleymani Fall 2016
2 Topics Probabilistic approach Bayes decision theory Generative models Gaussian Bayes classifier Naïve Bayes Discriminative models Logistic regression 2
3 Classification problem: probabilistic view Each feature as a random variable Class label also as a random variable We observe the feature values for a random sample and we intend to find its class label Evidence: feature vector x Query: class label 3
4 Definitions Posterior probability: p C k x Likelihood or class conditional probability: p x C k Prior probability: p(c k ) K p(x): pdf of feature vector x (p x = k=1 p x C k p(c k )) p(x C k ): pdf of feature vector x for samples of class C k p(c k ): probability of the label be C k 4
5 Bayes decision rule If P C 1 x > P(C 2 x) decide C 1 otherwise decide C 2 K = 2 p error x = p(c 2 x) if we decide C 1 P(C 1 x) if we decide C 2 If we use Bayes decision rule: P error x = min{p C 1 x, P(C 2 x)} Using Bayes rule, for each x, P error x is as small as possible and thus this rule minimizes the probability of error 5
6 Optimal classifier The optimal decision is the one that minimizes the expected number of mistakes We show that Bayes classifier is an optimal classifier 6
7 Bayes decision rule Minimizing misclassification rate Decision regions: R k = {x α x = k} K = 2 All points in R k are assigned to class C k p error = E x,y I(α(x) y) = p x R 1, C 2 + p x R 2, C 1 = R 1 p x, C 2 dx + R 2 p x, C 1 dx = R 1 p C 2 x p x dx + R 2 p C 1 x p x dx Choose class with highest p C k x as α x 7
8 Bayes minimum error Bayes minimum error classifier: min α(.) E x,y I(α(x) y) Zero-one loss If we know the probabilities in advance then the above optimization problem will be solved easily. α x = argmax y p(y x) In practice, we can estimate p(y x) based on a set of training samples D 8
9 Bayes theorem Bayes theorem Posterior Likelihood p C k x = p x C k p(c k ) p(x) Prior Posterior probability: p C k x Likelihood or class conditional probability: p x C k Prior probability: p(c k ) p(x): pdf of feature vector x (p x = k=1 K p x C k p(c k )) p(x C k ): pdf of feature vector x for samples of class C k p(c k ): probability of the label be C k 9
10 Bayes decision rule: example Bayes decision: Choose the class with highest p C k x p(c 1 x) p(x C 2 ) p(x C 1 ) p(c 2 x) R 2 R 2 p C 1 = 2 3 p C 2 = 1 3 p C k x = p x C k p(c k ) p(x) p x = p C 1 p x C 1 + p C 2 p x C 2 10
11 Bayesian decision rule If P C 1 x > P(C 2 x) decide C 1 otherwise decide C 2 Equivalent If p x C 1 P(C 1 ) p(x) > p x C 2 P(C 2 ) p(x) decide C 1 otherwise decide C 2 Equivalent If p x C 1 P(C 1 ) > p x C 2 P(C 2 ) decide C 1 otherwise decide C 2 11
12 Bayes decision rule: example Bayes decision: Choose the class with highest p C k x 2 p(x C 1 ) p(x C 2 ) p(x C 1 ) R 2 R 2 p C 1 = 2 3 p(c 1 x) p C 2 = 1 3 p(c 2 x) 12 R 2 R 2
13 Bayes Classier Simple Bayes classifier: estimate posterior probability of each class What should the decision criterion be? Choose class with highest p C k x The optimal decision is the one that minimizes the expected number of mistakes 13
14 Diabetes example white blood cell count 14 This example has been adopted from Sanja Fidler s slides, University of Toronto, CSC411
15 Diabetes example Doctor has a prior p y = 1 = 0.2 Prior: In the absence of any observation, what do I know about the probability of the classes? A patient comes in with white blood cell count x Does the patient have diabetes p y = 1 x? given a new observation, we still need to compute the posterior 15
16 Diabetes example p x y = 0 p x y = 1 16 This example has been adopted from Sanja Fidler s slides, University of Toronto, CSC411
17 Estimate probability densities from data If we assume Gaussian distributions for p(x C 1 ) and p(x C 2 ) Recall that for samples {x 1,, x N }, if we assume a Gaussian distribution, the MLE estimates will be 17
18 Diabetes example p x y = 0 p x y = 1 p x y = 1 = N μ 1, σ 1 2 μ 1 = n: y(n) =1 x(n) n: y (n) =1 σ 1 2 = n: y(n) =1 x n μ 1 N 1 n: y(n)=1 x(n) = 1 N This example has been adopted from Sanja Fidler s slides, University of Toronto, CSC411
19 Diabetes example Add a second observation: Plasma glucose value 19 This example has been adopted from Sanja Fidler s slides, University of Toronto, CSC411
20 Generative approach for this example Multivariate Gaussian distributions for p(x C k ): p x y = k 1 = 2π d/2 Σ 1/2 exp{ 1 2 x μ k T Σ 1 k x μ k } k = 1,2 Prior distribution p(x C k ): p y = 1 = π, p y = 0 = 1 π 20
21 MLE for multivariate Gaussian For samples {x 1,, x N }, if we assume a multivariate Gaussian distribution, the MLE estimates will be: μ = n=1 N N x (n) Σ = 1 N N n=1 x (n) μ x (n) μ T 21
22 Generative approach: example Maximum likelihood estimation (D = x n, y n N n=1 ): π = N 1 N μ 1 = Σ 1 = 1 N 1 Σ 2 = 1 N 2 n=1 N y (n) x (n) N N 1, μ 2 = n=1 (1 y (n) )x (n) N 2 n=1 N y (n) x (n) μ x (n) μ T n=1 N (1 y n ) x (n) μ x (n) μ T N 1 = N y (n) n=1 N 2 = N N 1 22
23 Decision boundary for Gaussian Bayes classifier p C 1 x = p(c 2 x) ln p(c 1 x) = ln p(c 2 x) ln p(x C 1 ) + ln p(c 1 ) ln p(x) = ln p(x C 2 ) + ln p(c 2 ) ln p(x) ln p(x C 1 ) + ln p(c 1 ) = ln p(x C 2 ) + ln p(c 2 ) ln p(x C k ) p C k x = p x C k p(c k ) p(x) = d 2 ln 2π 1 2 ln Σ k x μ k T Σ k 1 x μ k 23
24 Decision boundary p(x C 1 ) p(x C 2 ) p(c 1 x)=p(c 2 x) p(c 1 x) 24
25 Shared covariance matrix When classes share a single covariance matrix Σ = Σ 1 = Σ 2 p x C k = 1 2π d/2 Σ 1/2 exp{ 1 2 x μ k T Σ 1 x μ k } k = 1,2 p C 1 = π, p C 2 = 1 π 26
26 Likelihood N n=1 N p(x n, y (n) π, μ 1, μ 2, Σ) = p(x n y n, μ 1, μ 2, Σ)p(y n π) n=1 27
27 Shared covariance matrix Maximum likelihood estimation (D = x i, y i n i=1 ): μ 2 = μ 1 = π = N 1 N y (n) x (n) n=1 N n=1 N N 1 (1 y (n) )x (n) N 2 Σ = 1 N n C1 x (n) μ 1 x (n) μ 1 T + n C 2 x (n) μ 2 x (n) μ 2 T 28
28 Decision boundary when shared covariance matrix ln p(x C 1 ) + ln p(c 1 ) = ln p(x C 2 ) + ln p(c 2 ) ln p(x C k ) = d 2 ln 2π 1 2 ln Σ k x μ k T Σ 1 x μ k 29
29 Bayes decision rule Multi-class misclassification rate Multi-class problem: Probability of error of Bayesian decision rule Simpler to compute the probability of correct decision P error = 1 P(correct) P Correct = = K i=1 K i=1 R i p(x, C i ) dx R i p C i x p(x) dx R i : the subset of feature space assigned to the class C i using the classifier 31
30 Bayes minimum error Bayes minimum error classifier: min α(.) E x,y I(α(x) y) Zero-one loss α x = argmax y p(y x) 32
31 Minimizing Bayes risk (expected loss) K E x,y L α x, y = L α x, C j p x, C j dx j=1 K = p x L α x, C j p C j x dx j=1 for each x minimize it that is called conditional risk Bayes minimum loss (risk) decision rule: α(x) K α(x) = argmin i=1,,k j=1 L ij p C j x 33 The loss of assigning a sample to C i where the correct class is C j
32 Minimizing expected loss: special case (loss = misclassification rate) Problem definition for this special case: If action α x = i is taken and the true category is C j, then the decision is correct if i = j and otherwise it is incorrect. Zero-one loss function: L ij = 1 δ ij = 0 i = j 1 o. w. K α x = argmin i=1,,k j=1 L ij p C j x = argmin i=1,,k 0 p C i x + j i p C j x 35 = argmin i=1,,k 1 p C i x = argmax i=1,,k p C i x
33 Probabilistic discriminant functions Discriminant functions: A popular way of representing a classifier A discriminant function f i x for each class C i (i = 1,, K): x is assigned to class C i if: f i (x) > f j (x) j i Representing Bayesian classifier using discriminant functions: Classifier minimizing error rate: f i x = P(C i x) Classifier minimizing risk: f i x = K j=1 L ij p C j x 36
34 Naïve Bayes classifier Generative methods High number of parameters Assumption: Conditional independence p x C k = p x 1 C k p x 2 C k p x d C k 37
35 Naïve Bayes classifier In the decision phase, it finds the label of x according to: argmax k=1,,k argmax k=1,,k p(c k ) p C k x n i=1 p(x i C k ) p x C k = p x 1 C k p x 2 C k p x d C k p C k x p(c k ) p(x i C k ) n i=1 38
36 Naïve Bayes classifier Finds d univariate distributions p x 1 C k,, p x d C k of finding one multi-variate distribution p x C k instead Example 1: For Gaussian class-conditional density p x C k, it finds d + d (mean and sigma parameters on different dimensions) instead of d + d(d+1) parameters 2 Example 2: For Bernoulli class-conditional density p x C k, it finds d (mean parameters on different dimensions) instead of 2 d 1 parameters It first estimates the class conditional densities p x 1 C k,, p x d C k and the prior probability p(c k ) for each class (k = 1,, K) based on the training set. 39
37 Naïve Bayes: discrete example p H = Yes = 0.3 p D = Yes H = Yes = 1 3 p S = Yes H = Yes = 2 3 p D = Yes H = No = 2 7 p S = Yes H = No = 2 7 Decision on x = [Yes, Yes] (a person that has diabetes and also smokes): p H = Yes x p H = Yes p D = yes H = Yes p S = yes H = Yes = p H = No x p H = No p D = yes H = No p S = yes H = No = Thus decide H = yes Diabetes (D) Smoke (S) Heart Disease (H) Y N Y Y N N N Y N N Y N N N N N Y Y N N N N Y Y N N N Y N N 40
38 Probabilistic classifiers How can we find the probabilities required in the Bayes decision rule? Probabilistic classification approaches can be divided in two main categories: Generative Estimate pdf p(x, C k ) for each class C k p(c k x) or alternatively estimate both pdf p(x C k ) and p C k Discriminative Directly estimate p(c k x) for each class C k and then use it to find to find p(c k x) 41
39 Generative approach Inference stage Determine class conditional densities p(x C k ) p(c k ) Use the Bayes theorem to find p(c k x) and priors Decision stage: After learning the model (inference stage), make optimal class assignment for new input if p C i x > p C j x j i then decide C i 42
40 Discriminative vs. generative approach 43 [Bishop]
41 Class conditional densities vs. posterior p x C 1 p x C 2 p C 1 x [Bishop] p C 1 x = σ(w T x + w 0 ) 44 w = Σ 1 μ 1 μ 2 w 0 = 1 2 μ 1 T Σ 1 μ μ 2 T Σ 1 μ 2 + ln p(c 1) p(c 2 )
42 Discriminative approach Inference stage Determine the posterior class probabilities P(C k x) directly Decision stage: After learning the model (inference stage), make optimal class assignment for new input if P C i x > P C j x j i then decide C i 45
43 Posterior probabilities Two-class: p C k x can be written as a logistic sigmoid for a wide choice of p x C k distributions 1 p C 1 x = σ a(x) = 1 + exp( a(x)) Multi-class: p C k x can be written as a soft-max for a wide choice of p x C k p C k x = exp(a k(x)) exp(a j (x)) j=1 K 46
44 Discriminative approach: logistic regression More general than discriminant functions: f x; w predicts posterior probabilities P y = 1 x K = 2 f x; w = σ(w T x) x = 1, x 1,, x d w = w 0, w 1,, w d σ. is an activation function Sigmoid (logistic) function Activation function σ z = e z 47
45 Logistic regression f x; w : probability that y = 1 given x (parameterized by w) P y = 1 x, w = f x; w K = 2 y {0,1} P y = 0 x, w = 1 f x; w f x; w = σ(w T x) 0 f x; w 1 estimated probability of y = 1 on input x Example: Cancer (Malignant, Benign) f x; w = % chance of tumor being malignant 48
46 Logistic regression: Decision surface Decision surface f x; w = constant f x; w = σ w T 1 x = = e (wt x) Decision surfaces are linear functions of x if f x; w 0.5 then y = 1 else y = 0 Equivalent to if w T x + w 0 0 then y = 1 else y = 0 49
47 Logistic regression: ML estimation Maximum (conditional) log likelihood: n w = argmax w log i=1 p y (i) w, x (i) p y (i) w, x (i) = f x (i) ; w y(i) 1 f x (i) ; w (1 y(i) ) log p y X, w n = y (i) log f x (i) ; w i=1 + (1 y (i) )log 1 f x (i) ; w 50
48 Logistic regression: cost function w = argmin w J(w) J w = n n i=1 log p y (i) w, x (i) = y (i) log f x (i) ; w (1 y (i) )log 1 f x (i) ; w i=1 No closed form solution for However J(w) is convex. w J(w) = 0 51
49 Logistic regression: Gradient descent w t+1 = w t η w J(w t ) w J w = n i=1 f x i ; w y i x i Is it similar to gradient of SSE for linear regression? w J w = n i=1 w T x i y i x i 52
50 Logistic regression: loss function Loss y, f x; w = y log f x; w (1 y) log(1 f x; w ) Since y = 1 or y = 0 Loss y, f x; w = log(f(x; w)) if y = 1 log(1 f x; w ) if y = 0 How is it related to zero-one loss? Loss y, y = 1 y y 0 y = y f x; w = exp( w T x) 53
51 Logistic regression: cost function (summary) Logistic Regression (LR) has a more proper cost function for classification than SSE and Perceptron Why is the cost function of LR also more suitable than? J w = 1 n i=1 where f x; w = σ(w T x) n y i f x i ; w 2 The conditional distribution p y x, w in the classification problem is not Gaussian (it is Bernoulli) The cost function of LR is also convex 54
52 Multi-class logistic regression For each class k, f k x; W predicts the probability of y = k i.e., P(y = k x, W) On a new input x, to make a prediction, pick the class that maximizes f k x; W : α x = argmax k=1,,k f k x if f k x > f j x decide C k j k then 55
53 Multi-class logistic regression K > 2 y {1,2,, K} f k x; W = p y = k x = exp(w k T x ) K exp(w T j x ) j=1 Normalized exponential (aka softmax) If w k T x w j T x for all j k then p(c k x) 1, p(c j x) 0 56 p C k x = p x C k p(c k ) K p x C j p(c j ) j=1
54 Logistic regression: multi-class 57 J W = log = log = n i=1 W = argmin W n i=1 K k=1 n i=1 K k=1 p y i J(W) x i, W f k x i ; W y i k y k i log f k x (i) ; W y is a vector of length K (1-of-K coding) e.g., y = 0,0,1,0 T when the target class is C 3 W = w 1 w K Y = y (1) y (n) = y 1 (1) (1) y K (n) y K y 1 (n)
55 Logistic regression: multi-class w j t+1 = w j t η W J(W t ) wj J W = n i=1 f j x i ; W y j i x i 58
56 Logistic Regression (LR): summary LR is a linear classifier LR optimization problem is obtained by maximum likelihood when assuming Bernoulli distribution for conditional 1 probabilities whose mean is 1+e (wt x) No closed-form solution for its optimization problem But convex cost function and global optimum can be found by gradient ascent 59
57 Discriminative vs. generative: number of parameters d-dimensional feature space Logistic regression: d + 1 parameters w = (w 0, w 1,.., w d ) Generative approach: Gaussian class-conditionals with shared covariance matrix 2d parameters for means d(d + 1)/2 parameters for shared covariance matrix one parameter for class prior p(c 1 ). But LR is more robust, less sensitive to incorrect modeling assumptions 60
58 Summary of alternatives Generative Most demanding, because it finds the joint distribution p(x, C k ) Usually needs a large training set to find p(x C k ) Can find p(x) Outlier or novelty detection Discriminative Specifies what is really needed (i.e., p(c k x)) More computationally efficient 61
59 Resources C. Bishop, Pattern Recognition and Machine Learning, Chapter
Classification CE-717: Machine Learning Sharif University of Technology. M. Soleymani Fall 2012
Classification CE-717: Machine Learning Sharif University of Technology M. Soleymani Fall 2012 Topics Discriminant functions Logistic regression Perceptron Generative models Generative vs. discriminative
More informationCSC 411: Lecture 09: Naive Bayes
CSC 411: Lecture 09: Naive Bayes Class based on Raquel Urtasun & Rich Zemel s lectures Sanja Fidler University of Toronto Feb 8, 2015 Urtasun, Zemel, Fidler (UofT) CSC 411: 09-Naive Bayes Feb 8, 2015 1
More informationGaussian and Linear Discriminant Analysis; Multiclass Classification
Gaussian and Linear Discriminant Analysis; Multiclass Classification Professor Ameet Talwalkar Slide Credit: Professor Fei Sha Professor Ameet Talwalkar CS260 Machine Learning Algorithms October 13, 2015
More informationMachine Learning Basics Lecture 2: Linear Classification. Princeton University COS 495 Instructor: Yingyu Liang
Machine Learning Basics Lecture 2: Linear Classification Princeton University COS 495 Instructor: Yingyu Liang Review: machine learning basics Math formulation Given training data x i, y i : 1 i n i.i.d.
More informationLinear Models for Classification
Linear Models for Classification Oliver Schulte - CMPT 726 Bishop PRML Ch. 4 Classification: Hand-written Digit Recognition CHINE INTELLIGENCE, VOL. 24, NO. 24, APRIL 2002 x i = t i = (0, 0, 0, 1, 0, 0,
More informationMachine Learning Lecture 5
Machine Learning Lecture 5 Linear Discriminant Functions 26.10.2017 Bastian Leibe RWTH Aachen http://www.vision.rwth-aachen.de leibe@vision.rwth-aachen.de Course Outline Fundamentals Bayes Decision Theory
More informationLinear Classification: Probabilistic Generative Models
Linear Classification: Probabilistic Generative Models Sargur N. University at Buffalo, State University of New York USA 1 Linear Classification using Probabilistic Generative Models Topics 1. Overview
More informationLogistic Regression. Machine Learning Fall 2018
Logistic Regression Machine Learning Fall 2018 1 Where are e? We have seen the folloing ideas Linear models Learning as loss minimization Bayesian learning criteria (MAP and MLE estimation) The Naïve Bayes
More informationLogistic Regression Review Fall 2012 Recitation. September 25, 2012 TA: Selen Uguroglu
Logistic Regression Review 10-601 Fall 2012 Recitation September 25, 2012 TA: Selen Uguroglu!1 Outline Decision Theory Logistic regression Goal Loss function Inference Gradient Descent!2 Training Data
More informationCheng Soon Ong & Christian Walder. Canberra February June 2018
Cheng Soon Ong & Christian Walder Research Group and College of Engineering and Computer Science Canberra February June 2018 (Many figures from C. M. Bishop, "Pattern Recognition and ") 1of 305 Part VII
More informationMachine Learning Basics Lecture 7: Multiclass Classification. Princeton University COS 495 Instructor: Yingyu Liang
Machine Learning Basics Lecture 7: Multiclass Classification Princeton University COS 495 Instructor: Yingyu Liang Example: image classification indoor Indoor outdoor Example: image classification (multiclass)
More informationLogistic Regression. Jia-Bin Huang. Virginia Tech Spring 2019 ECE-5424G / CS-5824
Logistic Regression Jia-Bin Huang ECE-5424G / CS-5824 Virginia Tech Spring 2019 Administrative Please start HW 1 early! Questions are welcome! Two principles for estimating parameters Maximum Likelihood
More informationClassification Based on Probability
Logistic Regression These slides were assembled by Byron Boots, with only minor modifications from Eric Eaton s slides and grateful acknowledgement to the many others who made their course materials freely
More informationLogistic Regression. Seungjin Choi
Logistic Regression Seungjin Choi Department of Computer Science and Engineering Pohang University of Science and Technology 77 Cheongam-ro, Nam-gu, Pohang 37673, Korea seungjin@postech.ac.kr http://mlg.postech.ac.kr/
More informationMachine Learning 2017
Machine Learning 2017 Volker Roth Department of Mathematics & Computer Science University of Basel 21st March 2017 Volker Roth (University of Basel) Machine Learning 2017 21st March 2017 1 / 41 Section
More informationNaïve Bayes classification
Naïve Bayes classification 1 Probability theory Random variable: a variable whose possible values are numerical outcomes of a random phenomenon. Examples: A person s height, the outcome of a coin toss
More informationCh 4. Linear Models for Classification
Ch 4. Linear Models for Classification Pattern Recognition and Machine Learning, C. M. Bishop, 2006. Department of Computer Science and Engineering Pohang University of Science and echnology 77 Cheongam-ro,
More informationMachine Learning. Regression-Based Classification & Gaussian Discriminant Analysis. Manfred Huber
Machine Learning Regression-Based Classification & Gaussian Discriminant Analysis Manfred Huber 2015 1 Logistic Regression Linear regression provides a nice representation and an efficient solution to
More informationIntroduction to Machine Learning
Introduction to Machine Learning Logistic Regression Varun Chandola Computer Science & Engineering State University of New York at Buffalo Buffalo, NY, USA chandola@buffalo.edu Chandola@UB CSE 474/574
More informationParametric Models. Dr. Shuang LIANG. School of Software Engineering TongJi University Fall, 2012
Parametric Models Dr. Shuang LIANG School of Software Engineering TongJi University Fall, 2012 Today s Topics Maximum Likelihood Estimation Bayesian Density Estimation Today s Topics Maximum Likelihood
More informationOutline. Supervised Learning. Hong Chang. Institute of Computing Technology, Chinese Academy of Sciences. Machine Learning Methods (Fall 2012)
Outline Hong Chang Institute of Computing Technology, Chinese Academy of Sciences Machine Learning Methods (Fall 2012) Outline Outline I 1 Linear Models for Regression Linear Regression Probabilistic Interpretation
More informationNaïve Bayes classification. p ij 11/15/16. Probability theory. Probability theory. Probability theory. X P (X = x i )=1 i. Marginal Probability
Probability theory Naïve Bayes classification Random variable: a variable whose possible values are numerical outcomes of a random phenomenon. s: A person s height, the outcome of a coin toss Distinguish
More informationSupport Vector Machines
Support Vector Machines Le Song Machine Learning I CSE 6740, Fall 2013 Naïve Bayes classifier Still use Bayes decision rule for classification P y x = P x y P y P x But assume p x y = 1 is fully factorized
More informationIntroduction to Machine Learning
Introduction to Machine Learning Bayesian Classification Varun Chandola Computer Science & Engineering State University of New York at Buffalo Buffalo, NY, USA chandola@buffalo.edu Chandola@UB CSE 474/574
More informationLogistic Regression. Robot Image Credit: Viktoriya Sukhanova 123RF.com
Logistic Regression These slides were assembled by Eric Eaton, with grateful acknowledgement of the many others who made their course materials freely available online. Feel free to reuse or adapt these
More informationECE521 Lecture7. Logistic Regression
ECE521 Lecture7 Logistic Regression Outline Review of decision theory Logistic regression A single neuron Multi-class classification 2 Outline Decision theory is conceptually easy and computationally hard
More informationCPSC 340: Machine Learning and Data Mining. MLE and MAP Fall 2017
CPSC 340: Machine Learning and Data Mining MLE and MAP Fall 2017 Assignment 3: Admin 1 late day to hand in tonight, 2 late days for Wednesday. Assignment 4: Due Friday of next week. Last Time: Multi-Class
More informationIntroduction to Machine Learning
Outline Introduction to Machine Learning Bayesian Classification Varun Chandola March 8, 017 1. {circular,large,light,smooth,thick}, malignant. {circular,large,light,irregular,thick}, malignant 3. {oval,large,dark,smooth,thin},
More informationLINEAR CLASSIFICATION, PERCEPTRON, LOGISTIC REGRESSION, SVC, NAÏVE BAYES. Supervised Learning
LINEAR CLASSIFICATION, PERCEPTRON, LOGISTIC REGRESSION, SVC, NAÏVE BAYES Supervised Learning Linear vs non linear classifiers In K-NN we saw an example of a non-linear classifier: the decision boundary
More informationCSC 411: Lecture 04: Logistic Regression
CSC 411: Lecture 04: Logistic Regression Raquel Urtasun & Rich Zemel University of Toronto Sep 23, 2015 Urtasun & Zemel (UofT) CSC 411: 04-Prob Classif Sep 23, 2015 1 / 16 Today Key Concepts: Logistic
More informationChapter 3: Maximum-Likelihood & Bayesian Parameter Estimation (part 1)
HW 1 due today Parameter Estimation Biometrics CSE 190 Lecture 7 Today s lecture was on the blackboard. These slides are an alternative presentation of the material. CSE190, Winter10 CSE190, Winter10 Chapter
More informationBayesian Learning (II)
Universität Potsdam Institut für Informatik Lehrstuhl Maschinelles Lernen Bayesian Learning (II) Niels Landwehr Overview Probabilities, expected values, variance Basic concepts of Bayesian learning MAP
More informationUniversität Potsdam Institut für Informatik Lehrstuhl Maschinelles Lernen. Bayesian Learning. Tobias Scheffer, Niels Landwehr
Universität Potsdam Institut für Informatik Lehrstuhl Maschinelles Lernen Bayesian Learning Tobias Scheffer, Niels Landwehr Remember: Normal Distribution Distribution over x. Density function with parameters
More informationDensity Estimation: ML, MAP, Bayesian estimation
Density Estimation: ML, MAP, Bayesian estimation CE-725: Statistical Pattern Recognition Sharif University of Technology Spring 2013 Soleymani Outline Introduction Maximum-Likelihood Estimation Maximum
More informationLogistic Regression. Vibhav Gogate The University of Texas at Dallas. Some Slides from Carlos Guestrin, Luke Zettlemoyer and Dan Weld.
Logistic Regression Vibhav Gogate The University of Texas at Dallas Some Slides from Carlos Guestrin, Luke Zettlemoyer and Dan Weld. Generative vs. Discriminative Classifiers Want to Learn: h:x Y X features
More informationGenerative v. Discriminative classifiers Intuition
Logistic Regression Machine Learning 070/578 Carlos Guestrin Carnegie Mellon University September 24 th, 2007 Generative v. Discriminative classifiers Intuition Want to Learn: h:x a Y X features Y target
More informationBayesian Decision Theory
Bayesian Decision Theory Dr. Shuang LIANG School of Software Engineering TongJi University Fall, 2012 Today s Topics Bayesian Decision Theory Bayesian classification for normal distributions Error Probabilities
More informationStatistical Data Mining and Machine Learning Hilary Term 2016
Statistical Data Mining and Machine Learning Hilary Term 2016 Dino Sejdinovic Department of Statistics Oxford Slides and other materials available at: http://www.stats.ox.ac.uk/~sejdinov/sdmml Naïve Bayes
More informationLogistic Regression. William Cohen
Logistic Regression William Cohen 1 Outline Quick review classi5ication, naïve Bayes, perceptrons new result for naïve Bayes Learning as optimization Logistic regression via gradient ascent Over5itting
More informationMachine Learning Lecture 7
Course Outline Machine Learning Lecture 7 Fundamentals (2 weeks) Bayes Decision Theory Probability Density Estimation Statistical Learning Theory 23.05.2016 Discriminative Approaches (5 weeks) Linear Discriminant
More informationLast Time. Today. Bayesian Learning. The Distributions We Love. CSE 446 Gaussian Naïve Bayes & Logistic Regression
CSE 446 Gaussian Naïve Bayes & Logistic Regression Winter 22 Dan Weld Learning Gaussians Naïve Bayes Last Time Gaussians Naïve Bayes Logistic Regression Today Some slides from Carlos Guestrin, Luke Zettlemoyer
More informationMachine Learning for Signal Processing Bayes Classification and Regression
Machine Learning for Signal Processing Bayes Classification and Regression Instructor: Bhiksha Raj 11755/18797 1 Recap: KNN A very effective and simple way of performing classification Simple model: For
More informationGenerative v. Discriminative classifiers Intuition
Logistic Regression Machine Learning 10701/15781 Carlos Guestrin Carnegie Mellon University September 24 th, 2007 1 Generative v. Discriminative classifiers Intuition Want to Learn: h:x a Y X features
More information> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 BASEL. Logistic Regression. Pattern Recognition 2016 Sandro Schönborn University of Basel
Logistic Regression Pattern Recognition 2016 Sandro Schönborn University of Basel Two Worlds: Probabilistic & Algorithmic We have seen two conceptual approaches to classification: data class density estimation
More informationLogistic Regression. COMP 527 Danushka Bollegala
Logistic Regression COMP 527 Danushka Bollegala Binary Classification Given an instance x we must classify it to either positive (1) or negative (0) class We can use {1,-1} instead of {1,0} but we will
More informationECE521 week 3: 23/26 January 2017
ECE521 week 3: 23/26 January 2017 Outline Probabilistic interpretation of linear regression - Maximum likelihood estimation (MLE) - Maximum a posteriori (MAP) estimation Bias-variance trade-off Linear
More informationMulticlass Logistic Regression
Multiclass Logistic Regression Sargur. Srihari University at Buffalo, State University of ew York USA Machine Learning Srihari Topics in Linear Classification using Probabilistic Discriminative Models
More informationSTA 4273H: Statistical Machine Learning
STA 4273H: Statistical Machine Learning Russ Salakhutdinov Department of Statistics! rsalakhu@utstat.toronto.edu! http://www.utstat.utoronto.ca/~rsalakhu/ Sidney Smith Hall, Room 6002 Lecture 3 Linear
More informationLogistic Regression. Sargur N. Srihari. University at Buffalo, State University of New York USA
Logistic Regression Sargur N. University at Buffalo, State University of New York USA Topics in Linear Classification using Probabilistic Discriminative Models Generative vs Discriminative 1. Fixed basis
More informationStochastic Gradient Descent
Stochastic Gradient Descent Machine Learning CSE546 Carlos Guestrin University of Washington October 9, 2013 1 Logistic Regression Logistic function (or Sigmoid): Learn P(Y X) directly Assume a particular
More informationIntro. ANN & Fuzzy Systems. Lecture 15. Pattern Classification (I): Statistical Formulation
Lecture 15. Pattern Classification (I): Statistical Formulation Outline Statistical Pattern Recognition Maximum Posterior Probability (MAP) Classifier Maximum Likelihood (ML) Classifier K-Nearest Neighbor
More informationMachine Learning, Fall 2012 Homework 2
0-60 Machine Learning, Fall 202 Homework 2 Instructors: Tom Mitchell, Ziv Bar-Joseph TA in charge: Selen Uguroglu email: sugurogl@cs.cmu.edu SOLUTIONS Naive Bayes, 20 points Problem. Basic concepts, 0
More informationInf2b Learning and Data
Inf2b Learning and Data Lecture 13: Review (Credit: Hiroshi Shimodaira Iain Murray and Steve Renals) Centre for Speech Technology Research (CSTR) School of Informatics University of Edinburgh http://www.inf.ed.ac.uk/teaching/courses/inf2b/
More informationMachine Learning Tom M. Mitchell Machine Learning Department Carnegie Mellon University. September 20, 2012
Machine Learning 10-601 Tom M. Mitchell Machine Learning Department Carnegie Mellon University September 20, 2012 Today: Logistic regression Generative/Discriminative classifiers Readings: (see class website)
More informationAn Introduction to Statistical and Probabilistic Linear Models
An Introduction to Statistical and Probabilistic Linear Models Maximilian Mozes Proseminar Data Mining Fakultät für Informatik Technische Universität München June 07, 2017 Introduction In statistical learning
More informationLecture 3: Pattern Classification
EE E6820: Speech & Audio Processing & Recognition Lecture 3: Pattern Classification 1 2 3 4 5 The problem of classification Linear and nonlinear classifiers Probabilistic classification Gaussians, mixtures
More informationIs the test error unbiased for these programs?
Is the test error unbiased for these programs? Xtrain avg N o Preprocessing by de meaning using whole TEST set 2017 Kevin Jamieson 1 Is the test error unbiased for this program? e Stott see non for f x
More informationMachine Learning. Bayesian Regression & Classification. Marc Toussaint U Stuttgart
Machine Learning Bayesian Regression & Classification learning as inference, Bayesian Kernel Ridge regression & Gaussian Processes, Bayesian Kernel Logistic Regression & GP classification, Bayesian Neural
More informationSupport Vector Machine (SVM) & Kernel CE-717: Machine Learning Sharif University of Technology. M. Soleymani Fall 2012
Support Vector Machine (SVM) & Kernel CE-717: Machine Learning Sharif University of Technology M. Soleymani Fall 2012 Linear classifier Which classifier? x 2 x 1 2 Linear classifier Margin concept x 2
More informationIntroduction to Logistic Regression and Support Vector Machine
Introduction to Logistic Regression and Support Vector Machine guest lecturer: Ming-Wei Chang CS 446 Fall, 2009 () / 25 Fall, 2009 / 25 Before we start () 2 / 25 Fall, 2009 2 / 25 Before we start Feel
More informationLinear & nonlinear classifiers
Linear & nonlinear classifiers Machine Learning Hamid Beigy Sharif University of Technology Fall 1394 Hamid Beigy (Sharif University of Technology) Linear & nonlinear classifiers Fall 1394 1 / 34 Table
More informationMachine Learning
Machine Learning 10-701 Tom M. Mitchell Machine Learning Department Carnegie Mellon University February 1, 2011 Today: Generative discriminative classifiers Linear regression Decomposition of error into
More informationLinear Classification
Linear Classification Lili MOU moull12@sei.pku.edu.cn http://sei.pku.edu.cn/ moull12 23 April 2015 Outline Introduction Discriminant Functions Probabilistic Generative Models Probabilistic Discriminative
More informationThe Bayes classifier
The Bayes classifier Consider where is a random vector in is a random variable (depending on ) Let be a classifier with probability of error/risk given by The Bayes classifier (denoted ) is the optimal
More informationMachine Learning. Lecture 4: Regularization and Bayesian Statistics. Feng Li. https://funglee.github.io
Machine Learning Lecture 4: Regularization and Bayesian Statistics Feng Li fli@sdu.edu.cn https://funglee.github.io School of Computer Science and Technology Shandong University Fall 207 Overfitting Problem
More informationDiscriminative Models
No.5 Discriminative Models Hui Jiang Department of Electrical Engineering and Computer Science Lassonde School of Engineering York University, Toronto, Canada Outline Generative vs. Discriminative models
More informationCPSC 340: Machine Learning and Data Mining
CPSC 340: Machine Learning and Data Mining MLE and MAP Original version of these slides by Mark Schmidt, with modifications by Mike Gelbart. 1 Admin Assignment 4: Due tonight. Assignment 5: Will be released
More informationMachine Learning Gaussian Naïve Bayes Big Picture
Machine Learning 10-701 Tom M. Mitchell Machine Learning Department Carnegie Mellon University January 27, 2011 Today: Naïve Bayes Big Picture Logistic regression Gradient ascent Generative discriminative
More informationProbabilistic generative models
Linear models for classification Francesco Corona Probabilistic discriminative models Models with linear decision boundaries arise from assumptions about the data In a generative approach to classification,
More informationIntroduction to Machine Learning
1, DATA11002 Introduction to Machine Learning Lecturer: Teemu Roos TAs: Ville Hyvönen and Janne Leppä-aho Department of Computer Science University of Helsinki (based in part on material by Patrik Hoyer
More informationσ(a) = a N (x; 0, 1 2 ) dx. σ(a) = Φ(a) =
Until now we have always worked with likelihoods and prior distributions that were conjugate to each other, allowing the computation of the posterior distribution to be done in closed form. Unfortunately,
More informationLINEAR MODELS FOR CLASSIFICATION. J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
LINEAR MODELS FOR CLASSIFICATION Classification: Problem Statement 2 In regression, we are modeling the relationship between a continuous input variable x and a continuous target variable t. In classification,
More informationLecture 5: Linear models for classification. Logistic regression. Gradient Descent. Second-order methods.
Lecture 5: Linear models for classification. Logistic regression. Gradient Descent. Second-order methods. Linear models for classification Logistic regression Gradient descent and second-order methods
More informationDiscriminative Models
No.5 Discriminative Models Hui Jiang Department of Electrical Engineering and Computer Science Lassonde School of Engineering York University, Toronto, Canada Outline Generative vs. Discriminative models
More informationClassification Logistic Regression
Announcements: Classification Logistic Regression Machine Learning CSE546 Sham Kakade University of Washington HW due on Friday. Today: Review: sub-gradients,lasso Logistic Regression October 3, 26 Sham
More informationPATTERN RECOGNITION AND MACHINE LEARNING
PATTERN RECOGNITION AND MACHINE LEARNING Chapter 1. Introduction Shuai Huang April 21, 2014 Outline 1 What is Machine Learning? 2 Curve Fitting 3 Probability Theory 4 Model Selection 5 The curse of dimensionality
More informationProbabilistic Graphical Models: MRFs and CRFs. CSE628: Natural Language Processing Guest Lecturer: Veselin Stoyanov
Probabilistic Graphical Models: MRFs and CRFs CSE628: Natural Language Processing Guest Lecturer: Veselin Stoyanov Why PGMs? PGMs can model joint probabilities of many events. many techniques commonly
More informationUniversity of Cambridge Engineering Part IIB Module 3F3: Signal and Pattern Processing Handout 2:. The Multivariate Gaussian & Decision Boundaries
University of Cambridge Engineering Part IIB Module 3F3: Signal and Pattern Processing Handout :. The Multivariate Gaussian & Decision Boundaries..15.1.5 1 8 6 6 8 1 Mark Gales mjfg@eng.cam.ac.uk Lent
More informationMachine learning comes from Bayesian decision theory in statistics. There we want to minimize the expected value of the loss function.
Bayesian learning: Machine learning comes from Bayesian decision theory in statistics. There we want to minimize the expected value of the loss function. Let y be the true label and y be the predicted
More informationMachine Learning Linear Classification. Prof. Matteo Matteucci
Machine Learning Linear Classification Prof. Matteo Matteucci Recall from the first lecture 2 X R p Regression Y R Continuous Output X R p Y {Ω 0, Ω 1,, Ω K } Classification Discrete Output X R p Y (X)
More informationUniversität Potsdam Institut für Informatik Lehrstuhl Maschinelles Lernen. Linear Classifiers. Blaine Nelson, Tobias Scheffer
Universität Potsdam Institut für Informatik Lehrstuhl Linear Classifiers Blaine Nelson, Tobias Scheffer Contents Classification Problem Bayesian Classifier Decision Linear Classifiers, MAP Models Logistic
More informationLecture 2: Simple Classifiers
CSC 412/2506 Winter 2018 Probabilistic Learning and Reasoning Lecture 2: Simple Classifiers Slides based on Rich Zemel s All lecture slides will be available on the course website: www.cs.toronto.edu/~jessebett/csc412
More informationMachine Learning
Machine Learning 10-601 Tom M. Mitchell Machine Learning Department Carnegie Mellon University February 2, 2015 Today: Logistic regression Generative/Discriminative classifiers Readings: (see class website)
More informationIntroduction to Machine Learning
1, DATA11002 Introduction to Machine Learning Lecturer: Antti Ukkonen TAs: Saska Dönges and Janne Leppä-aho Department of Computer Science University of Helsinki (based in part on material by Patrik Hoyer,
More informationSequence labeling. Taking collective a set of interrelated instances x 1,, x T and jointly labeling them
HMM, MEMM and CRF 40-957 Special opics in Artificial Intelligence: Probabilistic Graphical Models Sharif University of echnology Soleymani Spring 2014 Sequence labeling aking collective a set of interrelated
More informationPart 4: Conditional Random Fields
Part 4: Conditional Random Fields Sebastian Nowozin and Christoph H. Lampert Colorado Springs, 25th June 2011 1 / 39 Problem (Probabilistic Learning) Let d(y x) be the (unknown) true conditional distribution.
More informationMachine Learning for Signal Processing Bayes Classification
Machine Learning for Signal Processing Bayes Classification Class 16. 24 Oct 2017 Instructor: Bhiksha Raj - Abelino Jimenez 11755/18797 1 Recap: KNN A very effective and simple way of performing classification
More informationChris Bishop s PRML Ch. 8: Graphical Models
Chris Bishop s PRML Ch. 8: Graphical Models January 24, 2008 Introduction Visualize the structure of a probabilistic model Design and motivate new models Insights into the model s properties, in particular
More informationMachine Learning. 7. Logistic and Linear Regression
Sapienza University of Rome, Italy - Machine Learning (27/28) University of Rome La Sapienza Master in Artificial Intelligence and Robotics Machine Learning 7. Logistic and Linear Regression Luca Iocchi,
More informationMidterm Review CS 7301: Advanced Machine Learning. Vibhav Gogate The University of Texas at Dallas
Midterm Review CS 7301: Advanced Machine Learning Vibhav Gogate The University of Texas at Dallas Supervised Learning Issues in supervised learning What makes learning hard Point Estimation: MLE vs Bayesian
More informationMidterm Review CS 6375: Machine Learning. Vibhav Gogate The University of Texas at Dallas
Midterm Review CS 6375: Machine Learning Vibhav Gogate The University of Texas at Dallas Machine Learning Supervised Learning Unsupervised Learning Reinforcement Learning Parametric Y Continuous Non-parametric
More informationBias-Variance Tradeoff
What s learning, revisited Overfitting Generative versus Discriminative Logistic Regression Machine Learning 10701/15781 Carlos Guestrin Carnegie Mellon University September 19 th, 2007 Bias-Variance Tradeoff
More informationMachine Learning
Machine Learning 10-601 Tom M. Mitchell Machine Learning Department Carnegie Mellon University February 4, 2015 Today: Generative discriminative classifiers Linear regression Decomposition of error into
More informationComments. x > w = w > x. Clarification: this course is about getting you to be able to think as a machine learning expert
Logistic regression Comments Mini-review and feedback These are equivalent: x > w = w > x Clarification: this course is about getting you to be able to think as a machine learning expert There has to be
More informationBayesian Decision and Bayesian Learning
Bayesian Decision and Bayesian Learning Ying Wu Electrical Engineering and Computer Science Northwestern University Evanston, IL 60208 http://www.eecs.northwestern.edu/~yingwu 1 / 30 Bayes Rule p(x ω i
More informationLearning Bayesian network : Given structure and completely observed data
Learning Bayesian network : Given structure and completely observed data Probabilistic Graphical Models Sharif University of Technology Spring 2017 Soleymani Learning problem Target: true distribution
More informationNon-Bayesian Classifiers Part II: Linear Discriminants and Support Vector Machines
Non-Bayesian Classifiers Part II: Linear Discriminants and Support Vector Machines Selim Aksoy Department of Computer Engineering Bilkent University saksoy@cs.bilkent.edu.tr CS 551, Fall 2018 CS 551, Fall
More informationProbabilistic modeling. The slides are closely adapted from Subhransu Maji s slides
Probabilistic modeling The slides are closely adapted from Subhransu Maji s slides Overview So far the models and algorithms you have learned about are relatively disconnected Probabilistic modeling framework
More informationCS 195-5: Machine Learning Problem Set 1
CS 95-5: Machine Learning Problem Set Douglas Lanman dlanman@brown.edu 7 September Regression Problem Show that the prediction errors y f(x; ŵ) are necessarily uncorrelated with any linear function of
More information