Machine Learning Lecture 2


 Charity Richard
 1 years ago
 Views:
Transcription
1 Machine Perceptual Learning and Sensory Summer Augmented 15 Computing Many slides adapted from B. Schiele Machine Learning Lecture 2 Probability Density Estimation Bastian Leibe RWTH Aachen
2 Announcements Course webpage Slides will be made available on the webpage L2P electronic repository Exercises and supplementary materials will be posted on the L2P Please subscribe to the lecture on the Campus system! Important to get announcements and L2P access! 2
3 Announcements Exercise sheet 1 is now available on L2P Bayes decision theory Maximum Likelihood Kernel density estimation / knn Submit your results to Ishrat/Michael until evening of Work in teams (of up to 3 people) is encouraged Who is not part of an exercise team yet? 3
4 Course Outline Fundamentals (2 weeks) Bayes Decision Theory Probability Density Estimation Discriminative Approaches (5 weeks) Linear Discriminant Functions Support Vector Machines Ensemble Methods & Boosting Randomized Trees, Forests & Ferns Generative Models (4 weeks) Bayesian Networks Markov Random Fields 4
5 Topics of This Lecture Bayes Decision Theory Basic concepts Minimizing the misclassification rate Minimizing the expected loss Discriminant functions Probability Density Estimation General concepts Gaussian distribution Parametric Methods Maximum Likelihood approach Bayesian vs. Frequentist views on probability Bayesian Learning 5
6 Recap: Bayes Decision Theory Concepts Concept 1: Priors (a priori probabilities) pc k What we can tell about the probability before seeing the data. Example: C C 1 2 a b pc pc In general: Slide credit: Bernt Schiele pck k 1 6
7 Recap: Bayes Decision Theory Concepts Concept 2: Conditional probabilities Let x be a feature vector. p x Ck x measures/describes certain properties of the input. E.g. number of black pixels, aspect ratio, p(x C k ) describes its likelihood for class C k. p x a p x b x Slide credit: Bernt Schiele x 7
8 Bayes Decision Theory Concepts Concept 3: Posterior probabilities p Ck We are typically interested in the a posteriori probability, i.e. the probability of class C k given the measurement vector x. x Bayes Theorem: p C Interpretation k x p x C k p Ck p x Ck p Ck p x p x Ci p C Likelihood Prior Posterior Normalization Factor i i Slide credit: Bernt Schiele 8
9 Bayes Decision Theory p x a p x b Likelihood p x a p( a) x p x b p( b) x Decision boundary Likelihood Prior p a x p b x Slide credit: Bernt Schiele x Posterior = Likelihood Prior NormalizationFactor 9
10 Bayesian Decision Theory Goal: Minimize the probability of a misclassification The green and blue regions stay constant. Only the size of the red region varies! = Z R 1 p(c 2 jx)p(x)dx + Z p(c 1 jx)p(x)dx R 2 12 Image source: C.M. Bishop, 2006
11 Bayes Decision Theory Optimal decision rule Decide for C 1 if p(c 1 jx) > p(c 2 jx) This is equivalent to p(xjc 1 )p(c 1 ) > p(xjc 2 )p(c 2 ) Which is again equivalent to (LikelihoodRatio test) p(xjc 1 ) p(xjc 2 ) > p(c 2) p(c 1 ) Slide credit: Bernt Schiele Decision threshold 13
12 Generalization to More Than 2 Classes Decide for class k whenever it has the greatest posterior probability of all classes: p(c k jx) > p(c j jx) 8j 6= k p(xjc k )p(c k ) > p(xjc j )p(c j ) 8j 6= k Likelihoodratio test p(xjc k ) p(xjc j ) > p(c j) p(c k ) 8j 6= k Slide credit: Bernt Schiele 14
13 Bayes Decision Theory Decision regions: R 1, R 2, R 3, Slide credit: Bernt Schiele 15
14 Classifying with Loss Functions Generalization to decisions with a loss function Differentiate between the possible decisions and the possible true classes. Example: medical diagnosis Decisions: diagnosis is sick or healthy (or: further examination necessary) Classes: patient is sick or healthy The cost may be asymmetric: loss(decision = healthyjpatient = sick) >> loss(decision = sickjpatient = healthy) Slide credit: Bernt Schiele 16
15 Truth Classifying with Loss Functions In general, we can formalize this by introducing a loss matrix L kj L kj = loss for decision C j if truth is C k : Example: cancer diagnosis Decision L cancer diagnosis = 17
16 Classifying with Loss Functions Loss functions may be different for different actors. Example: L stocktrader (subprime) = invest don t invest µ 1 2 c gain L bank (subprime) = µ 1 2 c gain 0 0 Different loss functions may lead to different Bayes optimal strategies. 18
17 Minimizing the Expected Loss Optimal solution is the one that minimizes the loss. But: loss function depends on the true class, which is unknown. Solution: Minimize the expected loss This can be done by choosing the regions R j such that which is easy to do once we know the posterior class probabilities p(c k jx). 19
18 Minimizing the Expected Loss Example: 2 Classes: C 1, C 2 2 Decision: 1, 2 Loss function: L( j jc k ) = L kj Expected loss (= risk R) for the two decisions: Goal: Decide such that expected loss is minimized I.e. decide 1 if Slide credit: Bernt Schiele 20
19 Minimizing the Expected Loss R( 2 jx) > R( 1 jx) L 12 p(c 1 jx) + L 22 p(c 2 jx) > L 11 p(c 1 jx) + L 21 p(c 2 jx) (L 12 L 11 )p(c 1 jx) > (L 21 L 22 )p(c 2 jx) (L 12 L 11 ) (L 21 L 22 ) > p(c 2jx) p(c 1 jx) = p(xjc 2)p(C 2 ) p(xjc 1 )p(c 1 ) p(xjc 1 ) p(xjc 2 ) > (L 21 L 22 ) (L 12 L 11 ) p(c 2 ) p(c 1 ) Adapted decision rule taking into account the loss. Slide credit: Bernt Schiele 21
20 The Reject Option Classification errors arise from regions where the largest posterior probability p(c k jx) is significantly less than 1. These are the regions where we are relatively uncertain about class membership. For some applications, it may be better to reject the automatic decision entirely in such a case and e.g. consult a human expert. 22 Image source: C.M. Bishop, 2006
21 Discriminant Functions Formulate classification in terms of comparisons Discriminant functions y 1 (x); : : : ; y K (x) Classify x as class C k if y k (x) > y j (x) 8j 6= k Examples (Bayes Decision Theory) y k (x) = p(c k jx) y k (x) = p(xjc k )p(c k ) y k (x) = log p(xjc k ) + log p(c k ) Slide credit: Bernt Schiele 23
22 Different Views on the Decision Problem y k (x) / p(xjc k )p(c k ) First determine the classconditional densities for each class individually and separately infer the prior class probabilities. Then use Bayes theorem to determine class membership. Generative methods y k (x) = p(c k jx) First solve the inference problem of determining the posterior class probabilities. Then use decision theory to assign each new x to its class. Discriminative methods Alternative Directly find a discriminant function which maps each input x directly onto a class label. y k (x) 24
23 Topics of This Lecture Bayes Decision Theory Basic concepts Minimizing the misclassification rate Minimizing the expected loss Discriminant functions Probability Density Estimation General concepts Gaussian distribution Parametric Methods Maximum Likelihood approach Bayesian vs. Frequentist views on probability Bayesian Learning 25
24 Probability Density Estimation Up to now Bayes optimal classification Based on the probabilities p(xjc k )p(c k ) How can we estimate (=learn) those probability densities? Supervised training case: data and class labels are known. Estimate the probability density for each class separately: p(xjc k ) (For simplicity of notation, we will drop the class label in the following.) C k C k Slide credit: Bernt Schiele 26
25 Probability Density Estimation Data: x 1, x 2, x 3, x 4, x Estimate: p(x) x Methods Parametric representations Nonparametric representations Mixture models (next lecture) Slide credit: Bernt Schiele 27
26 The Gaussian (or Normal) Distribution Onedimensional case Mean ¹ Variance ¾ 2 N(xj¹; ¾ 2 ) = p 1 ¾ (x ¹)2 exp ½ 2¼¾ 2¾ 2 Multidimensional case Mean ¹ Covariance N(xj¹; ) = ½ 1 exp 1 ¾ (2¼) D=2 j j1=2 2 (x ¹)T 1 (x ¹) 28 Image source: C.M. Bishop, 2006
27 Gaussian Distribution Properties Central Limit Theorem The distribution of the sum of N i.i.d. random variables becomes increasingly Gaussian as N grows. In practice, the convergence to a Gaussian can be very rapid. This makes the Gaussian interesting for many applications. Example: N uniform [0,1] random variables. 29 Image source: C.M. Bishop, 2006
28 Gaussian Distribution Properties Quadratic Form N depends on x through the exponent Here, M is often called the Mahalanobis distance from ¹ to x. Shape of the Gaussian is a real, symmetric matrix. We can therefore decompose it into its eigenvectors DX = iu i u T i i=1 and thus obtain with. Constant density on ellipsoids with main directions along the eigenvectors u i and scaling factors p i. 30 Image source: C.M. Bishop, 2006
29 Gaussian Distribution Properties Special cases Full covariance matrix = [¾ ij ] General ellipsoid shape Diagonal covariance matrix = diagf¾ i g Axisaligned ellipsoid Uniform variance = ¾ 2 I Hypersphere 31 Image source: C.M. Bishop, 2006
30 Gaussian Distribution Properties The marginals of a Gaussian are again Gaussians: 32 Image source: C.M. Bishop, 2006
31 Topics of This Lecture Bayes Decision Theory Basic concepts Minimizing the misclassification rate Minimizing the expected loss Discriminant functions Probability Density Estimation General concepts Gaussian distribution Parametric Methods Maximum Likelihood approach Bayesian vs. Frequentist views on probability Bayesian Learning 33
32 Parametric Methods Given Data Parametric form of the distribution with parameters µ E.g. for Gaussian distrib.: Learning X = fx 1 ; x 2 ; : : : ; x N g Estimation of the parameters µ µ = (¹; ¾) x x Likelihood of µ Probability that the data X have indeed been generated from a probability density with parameters µ L(µ) = p(xjµ) Slide adapted from Bernt Schiele 34
33 Maximum Likelihood Approach Computation of the likelihood p(x n jµ) Single data point: Assumption: all data points are independent Loglikelihood L(µ) = p(xjµ) = E(µ) = ln L(µ) = NY n=1 p(x n jµ) NX ln p(x n jµ) n=1 Estimation of the parameters µ (Learning) Maximize the likelihood Minimize the negative loglikelihood Slide credit: Bernt Schiele 35
34 Maximum Likelihood Approach Likelihood: L(µ) = p(xjµ) = NY n=1 p(x n jµ) We want to obtain ^µ such that L(^µ) is maximized. p(xjµ) ^µ µ Slide credit: Bernt Schiele 36
35 Maximum Likelihood Approach Minimizing the loglikelihood How do we minimize a function? Take the derivative and set it NX ln p(x n jµ) = n=1 p(x njµ) p(x n jµ)! = 0 Loglikelihood for Normal distribution (1D case) NX E(µ) = ln p(x n j¹; ¾) = n=1 NX n=1 ln µ 1 p exp ½ jjx n ¹jj 2 ¾ 2¼¾ 2¾ 2 37
36 Maximum Likelihood Approach Minimizing E(¹; ¾) = p(x nj¹; ¾) p(x n j¹; ¾) = n=1 NX 2(x n ¹) 2¾ 2 n=1 X N = 1 NX 2 (x (x n n ¹) ¹) ¾ n=1 Ã = 1 N! X 2 x n n N¹ E(¹; ¾) =! 0, ^¹ = 1 N NX n=1 x n p(x n j¹; ¾) = 1 p e jjx n ¹jj 2 2¾ 2 2¼¾ 38
37 Maximum Likelihood Approach We thus obtain ^¹ = 1 NX N n=1 x n In a similar fashion, we get ^¾ 2 = 1 NX (x n ^¹) 2 N n=1 sample mean sample variance ^µ = (^¹; ^¾) is the Maximum Likelihood estimate for the parameters of a Gaussian distribution. This is a very important result. Unfortunately, it is wrong 39
38 Maximum Likelihood Approach Or not wrong, but rather biased Assume the samples x 1, x 2,, x N come from a true Gaussian distribution with mean ¹ and variance ¾ 2 We can now compute the expectations of the ML estimates with respect to the data set values. It can be shown that E(¹ ML ) = ¹ µ N 1 E(¾ML) 2 = N ¾ 2 The ML estimate will underestimate the true variance. Corrected estimate: ~¾ 2 = N N 1 ¾2 ML = 1 N 1 NX (x n ^¹) 2 n=1 40
39 Maximum Likelihood Limitations Maximum Likelihood has several significant limitations It systematically underestimates the variance of the distribution! E.g. consider the case N = 1; X = fx 1 g x Maximumlikelihood estimate: ^¾ = 0! We say ML overfits to the observed data. We will still often use ML, but it is important to know about this effect. ^¹ x Slide adapted from Bernt Schiele 41
40 Deeper Reason Maximum Likelihood is a Frequentist concept In the Frequentist view, probabilities are the frequencies of random, repeatable events. These frequencies are fixed, but can be estimated more precisely when more data is available. This is in contrast to the Bayesian interpretation In the Bayesian view, probabilities quantify the uncertainty about certain states or events. This uncertainty can be revised in the light of new evidence. Bayesians and Frequentists do not like each other too well 42
41 Bayesian vs. Frequentist View To see the difference Suppose we want to estimate the uncertainty whether the Arctic ice cap will have disappeared by the end of the century. This question makes no sense in a Frequentist view, since the event cannot be repeated numerous times. In the Bayesian view, we generally have a prior, e.g. from calculations how fast the polar ice is melting. If we now get fresh evidence, e.g. from a new satellite, we may revise our opinion and update the uncertainty from the prior. Posterior / Likelihood Prior This generally allows to get better uncertainty estimates for many situations. Main Frequentist criticism The prior has to come from somewhere and if it is wrong, the result will be worse. 43
42 Bayesian Approach to Parameter Learning Conceptual shift Maximum Likelihood views the true parameter vector µ to be unknown, but fixed. In Bayesian learning, we consider µ to be a random variable. This allows us to use knowledge about the parameters µ i.e. to use a prior for µ Training data then converts this prior distribution on µ into a posterior probability density. The prior thus encodes knowledge we have about the type of distribution we expect to see for µ. Slide adapted from Bernt Schiele 44
43 Bayesian Learning Approach Bayesian view: Consider the parameter vector µ as a random variable. When estimating the parameters, what we compute is p(xjx) = p(xjx) = Z Z p(x; µjx)dµ p(x; µjx) = p(xjµ; X)p(µjX) p(xjµ)p(µjx)dµ Assumption: given µ, this doesn t depend on X anymore This is entirely determined by the parameter µ (i.e. by the parametric form of the pdf). Slide adapted from Bernt Schiele 45
44 Bayesian Learning Approach Z p(xjx) = p(xjµ)p(µjx)dµ p(µjx) = p(xjµ)p(µ) p(x) p(x) = Z p(xjµ)p(µ)dµ = = p(µ) p(x) L(µ) Z L(µ)p(µ)dµ Inserting this above, we obtain Z p(xjµ)l(µ)p(µ) p(xjx) = dµ = p(x) Z p(xjµ)l(µ)p(µ) R L(µ)p(µ)dµ dµ Slide credit: Bernt Schiele 46
45 Bayesian Learning Approach Discussion Likelihood of the parametric form µ given the data set X. Estimate for x based on parametric form µ Prior for the parameters µ p(xjx) = Z p(xjµ)l(µ)p(µ) R L(µ)p(µ)dµ dµ Normalization: integrate over all possible values of µ If we now plug in a (suitable) prior p(µ), we can estimate from the data set X. p(xjx) 47
46 Bayesian Density Estimation Discussion p(xjx) = Z p(xjµ)p(µjx)dµ = Z p(xjµ)l(µ)p(µ) R L(µ)p(µ)dµ dµ p(µjx) The probability makes the dependency of the estimate on the data explicit. p(µjx) ^µ If is very small everywhere, but is large for one, then p(xjx) ¼ p(xj^µ) The more uncertain we are about µ, the more we average over all parameter values. Slide credit: Bernt Schiele 48
47 Bayesian Density Estimation Problem In the general case, the integration over µ is not possible (or only possible stochastically). Example where an analytical solution is possible Normal distribution for the data, ¾ 2 assumed known and fixed. Estimate the distribution of the mean: p(¹jx) = p(xj¹)p(¹) p(x) Prior: We assume a Gaussian prior over ¹, Slide credit: Bernt Schiele 49
48 Bayesian Learning Approach Sample mean: Bayes estimate: Note: ¹ N = ¾2 ¹ 0 + N¾ 2 0 ¹x ¾ 2 + N¾ ¾ 2 N = 1 ¾ N ¾ 2 ¹x = 1 N NX n=1 x n p(¹jx) ¹ 0 = 0 50 Slide adapted from Bernt Schiele Image source: C.M. Bishop, 2006
49 Summary: ML vs. Bayesian Learning Maximum Likelihood Simple approach, often analytically possible. Problem: estimation is biased, tends to overfit to the data. But: Often needs some correction or regularization. Approximation gets accurate for N! 1. Bayesian Learning General approach, avoids the estimation bias through a prior. Problems: But: Need to choose a suitable prior (not always obvious). Integral over µ often not analytically feasible anymore. Efficient stochastic sampling techniques available (see Lecture 15). (In this lecture, we ll use both concepts wherever appropriate) 51
50 References and Further Reading More information in Bishop s book Gaussian distribution and ML: Ch and Bayesian Learning: Ch and Nonparametric methods: Ch Additional information can be found in Duda & Hart ML estimation: Ch. 3.2 Bayesian Learning: Ch Nonparametric methods: Ch Christopher M. Bishop Pattern Recognition and Machine Learning Springer, 2006 R.O. Duda, P.E. Hart, D.G. Stork Pattern Classification 2 nd Ed., WileyInterscience,
Machine Learning Lecture 2
Machine Perceptual Learning and Sensory Summer Augmented 6 Computing Announcements Machine Learning Lecture 2 Course webpage http://www.vision.rwthaachen.de/teaching/ Slides will be made available on
More informationMachine Learning Lecture 2
Announcements Machine Learning Lecture 2 Eceptional number of lecture participants this year Current count: 449 participants This is very nice, but it stretches our resources to their limits Probability
More informationMachine Learning Lecture 1
Many slides adapted from B. Schiele Machine Learning Lecture 1 Introduction 18.04.2016 Bastian Leibe RWTH Aachen http://www.vision.rwthaachen.de/ leibe@vision.rwthaachen.de Organization Lecturer Prof.
More informationMachine Learning Lecture 3
Announcements Machine Learning Lecture 3 Eam dates We re in the process of fiing the first eam date Probability Density Estimation II 9.0.207 Eercises The first eercise sheet is available on L2P now First
More informationLecture Machine Learning (past summer semester) If at least one Englishspeaking student is present.
Advanced Machine Learning Lecture 1 Organization Lecturer Prof. Bastian Leibe (leibe@vision.rwthaachen.de) Introduction 20.10.2015 Bastian Leibe Visual Computing Institute RWTH Aachen University http://www.vision.rwthaachen.de/
More informationMachine Learning Lecture 5
Machine Learning Lecture 5 Linear Discriminant Functions 26.10.2017 Bastian Leibe RWTH Aachen http://www.vision.rwthaachen.de leibe@vision.rwthaachen.de Course Outline Fundamentals Bayes Decision Theory
More informationMachine Learning Lecture 7
Course Outline Machine Learning Lecture 7 Fundamentals (2 weeks) Bayes Decision Theory Probability Density Estimation Statistical Learning Theory 23.05.2016 Discriminative Approaches (5 weeks) Linear Discriminant
More informationLecture 3. Linear Regression II Bastian Leibe RWTH Aachen
Advanced Machine Learning Lecture 3 Linear Regression II 02.11.2015 Bastian Leibe RWTH Aachen http://www.vision.rwthaachen.de/ leibe@vision.rwthaachen.de This Lecture: Advanced Machine Learning Regression
More informationLecture 10. Announcement. Mixture Models II. Topics of This Lecture. This Lecture: Advanced Machine Learning. Recap: GMMs as Latent Variable Models
Advanced Machine Learning Lecture 10 Mixture Models II 30.11.2015 Bastian Leibe RWTH Aachen http://www.vision.rwthaachen.de/ Announcement Exercise sheet 2 online Sampling Rejection Sampling Importance
More informationECE521 week 3: 23/26 January 2017
ECE521 week 3: 23/26 January 2017 Outline Probabilistic interpretation of linear regression  Maximum likelihood estimation (MLE)  Maximum a posteriori (MAP) estimation Biasvariance tradeoff Linear
More informationMachine Learning Lecture 14
Many slides adapted from B. Schiele, S. Roth, Z. Gharahmani Machine Learning Lecture 14 Undirected Graphical Models & Inference 23.06.2015 Bastian Leibe RWTH Aachen http://www.vision.rwthaachen.de/ leibe@vision.rwthaachen.de
More informationParametric Models. Dr. Shuang LIANG. School of Software Engineering TongJi University Fall, 2012
Parametric Models Dr. Shuang LIANG School of Software Engineering TongJi University Fall, 2012 Today s Topics Maximum Likelihood Estimation Bayesian Density Estimation Today s Topics Maximum Likelihood
More informationCheng Soon Ong & Christian Walder. Canberra February June 2018
Cheng Soon Ong & Christian Walder Research Group and College of Engineering and Computer Science Canberra February June 2018 (Many figures from C. M. Bishop, "Pattern Recognition and ") 1of 143 Part IV
More informationMachine Learning Lecture 12
Machine Learning Lecture 12 Neural Networks 30.11.2017 Bastian Leibe RWTH Aachen http://www.vision.rwthaachen.de leibe@vision.rwthaachen.de Course Outline Fundamentals Bayes Decision Theory Probability
More informationChapter 3: MaximumLikelihood & Bayesian Parameter Estimation (part 1)
HW 1 due today Parameter Estimation Biometrics CSE 190 Lecture 7 Today s lecture was on the blackboard. These slides are an alternative presentation of the material. CSE190, Winter10 CSE190, Winter10 Chapter
More informationMultivariate statistical methods and data mining in particle physics
Multivariate statistical methods and data mining in particle physics RHUL Physics www.pp.rhul.ac.uk/~cowan Academic Training Lectures CERN 16 19 June, 2008 1 Outline Statement of the problem Some general
More informationStatistical and Learning Techniques in Computer Vision Lecture 2: Maximum Likelihood and Bayesian Estimation Jens Rittscher and Chuck Stewart
Statistical and Learning Techniques in Computer Vision Lecture 2: Maximum Likelihood and Bayesian Estimation Jens Rittscher and Chuck Stewart 1 Motivation and Problem In Lecture 1 we briefly saw how histograms
More information5. Discriminant analysis
5. Discriminant analysis We continue from Bayes s rule presented in Section 3 on p. 85 (5.1) where c i is a class, x isapdimensional vector (data case) and we use class conditional probability (density
More informationBayesian Decision Theory
Bayesian Decision Theory Dr. Shuang LIANG School of Software Engineering TongJi University Fall, 2012 Today s Topics Bayesian Decision Theory Bayesian classification for normal distributions Error Probabilities
More informationLecture 12. Neural Networks Bastian Leibe RWTH Aachen
Advanced Machine Learning Lecture 12 Neural Networks 10.12.2015 Bastian Leibe RWTH Aachen http://www.vision.rwthaachen.de/ leibe@vision.rwthaachen.de This Lecture: Advanced Machine Learning Regression
More informationInstancebased Learning CE717: Machine Learning Sharif University of Technology. M. Soleymani Fall 2016
Instancebased Learning CE717: Machine Learning Sharif University of Technology M. Soleymani Fall 2016 Outline Nonparametric approach Unsupervised: Nonparametric density estimation Parzen Windows KnNearest
More informationENG 8801/ Special Topics in Computer Engineering: Pattern Recognition. Memorial University of Newfoundland Pattern Recognition
Memorial University of Newfoundland Pattern Recognition Lecture 6 May 18, 2006 http://www.engr.mun.ca/~charlesr Office Hours: Tuesdays & Thursdays 8:309:30 PM EN3026 Review Distancebased Classification
More informationIntro. ANN & Fuzzy Systems. Lecture 15. Pattern Classification (I): Statistical Formulation
Lecture 15. Pattern Classification (I): Statistical Formulation Outline Statistical Pattern Recognition Maximum Posterior Probability (MAP) Classifier Maximum Likelihood (ML) Classifier KNearest Neighbor
More informationSYDE 372 Introduction to Pattern Recognition. Probability Measures for Classification: Part I
SYDE 372 Introduction to Pattern Recognition Probability Measures for Classification: Part I Alexander Wong Department of Systems Design Engineering University of Waterloo Outline 1 2 3 4 Why use probability
More informationBAYESIAN DECISION THEORY
Last updated: September 17, 2012 BAYESIAN DECISION THEORY Problems 2 The following problems from the textbook are relevant: 2.1 2.9, 2.11, 2.17 For this week, please at least solve Problem 2.3. We will
More informationPATTERN RECOGNITION AND MACHINE LEARNING
PATTERN RECOGNITION AND MACHINE LEARNING Chapter 1. Introduction Shuai Huang April 21, 2014 Outline 1 What is Machine Learning? 2 Curve Fitting 3 Probability Theory 4 Model Selection 5 The curse of dimensionality
More informationLecture 4: Probabilistic Learning. Estimation Theory. Classification with Probability Distributions
DD2431 Autumn, 2014 1 2 3 Classification with Probability Distributions Estimation Theory Classification in the last lecture we assumed we new: P(y) Prior P(x y) Lielihood x2 x features y {ω 1,..., ω K
More informationPattern Recognition and Machine Learning. Bishop Chapter 2: Probability Distributions
Pattern Recognition and Machine Learning Chapter 2: Probability Distributions Cécile Amblard Alex Kläser Jakob Verbeek October 11, 27 Probability Distributions: General Density Estimation: given a finite
More informationProblem Set 2. MAS 622J/1.126J: Pattern Recognition and Analysis. Due: 5:00 p.m. on September 30
Problem Set 2 MAS 622J/1.126J: Pattern Recognition and Analysis Due: 5:00 p.m. on September 30 [Note: All instructions to plot data or write a program should be carried out using Matlab. In order to maintain
More informationParametric Techniques
Parametric Techniques Jason J. Corso SUNY at Buffalo J. Corso (SUNY at Buffalo) Parametric Techniques 1 / 39 Introduction When covering Bayesian Decision Theory, we assumed the full probabilistic structure
More informationBayesian decision theory Introduction to Pattern Recognition. Lectures 4 and 5: Bayesian decision theory
Bayesian decision theory 8001652 Introduction to Pattern Recognition. Lectures 4 and 5: Bayesian decision theory Jussi Tohka jussi.tohka@tut.fi Institute of Signal Processing Tampere University of Technology
More informationMidterm. Introduction to Machine Learning. CS 189 Spring Please do not open the exam before you are instructed to do so.
CS 89 Spring 07 Introduction to Machine Learning Midterm Please do not open the exam before you are instructed to do so. The exam is closed book, closed notes except your onepage cheat sheet. Electronic
More informationCOMP 551 Applied Machine Learning Lecture 19: Bayesian Inference
COMP 551 Applied Machine Learning Lecture 19: Bayesian Inference Associate Instructor: (herke.vanhoof@mcgill.ca) Class web page: www.cs.mcgill.ca/~jpineau/comp551 Unless otherwise noted, all material posted
More informationCh 4. Linear Models for Classification
Ch 4. Linear Models for Classification Pattern Recognition and Machine Learning, C. M. Bishop, 2006. Department of Computer Science and Engineering Pohang University of Science and echnology 77 Cheongamro,
More informationIntelligent Systems Statistical Machine Learning
Intelligent Systems Statistical Machine Learning Carsten Rother, Dmitrij Schlesinger WS2014/2015, Our tasks (recap) The model: two variables are usually present:  the first one is typically discrete k
More informationCurve Fitting Revisited, Bishop1.2.5
Curve Fitting Revisited, Bishop1.2.5 Maximum Likelihood Bishop 1.2.5 Model Likelihood differentiation p(t x, w, β) = Maximum Likelihood N N ( t n y(x n, w), β 1). (1.61) n=1 As we did in the case of the
More informationParametric Techniques Lecture 3
Parametric Techniques Lecture 3 Jason Corso SUNY at Buffalo 22 January 2009 J. Corso (SUNY at Buffalo) Parametric Techniques Lecture 3 22 January 2009 1 / 39 Introduction In Lecture 2, we learned how to
More informationOverfitting, Bias / Variance Analysis
Overfitting, Bias / Variance Analysis Professor Ameet Talwalkar Professor Ameet Talwalkar CS260 Machine Learning Algorithms February 8, 207 / 40 Outline Administration 2 Review of last lecture 3 Basic
More informationLecture 12. Neural Networks Bastian Leibe RWTH Aachen
Advanced Machine Learning Lecture 12 Neural Networks 24.11.2016 Bastian Leibe RWTH Aachen http://www.vision.rwthaachen.de/ leibe@vision.rwthaachen.de Talk Announcement Yann LeCun (NYU & FaceBook AI)
More informationMachine Learning Lecture 10
Machine Learning Lecture 10 Neural Networks 26.11.2018 Bastian Leibe RWTH Aachen http://www.vision.rwthaachen.de leibe@vision.rwthaachen.de Today s Topic Deep Learning 2 Course Outline Fundamentals Bayes
More informationClassification CE717: Machine Learning Sharif University of Technology. M. Soleymani Fall 2012
Classification CE717: Machine Learning Sharif University of Technology M. Soleymani Fall 2012 Topics Discriminant functions Logistic regression Perceptron Generative models Generative vs. discriminative
More informationThe Bayes classifier
The Bayes classifier Consider where is a random vector in is a random variable (depending on ) Let be a classifier with probability of error/risk given by The Bayes classifier (denoted ) is the optimal
More informationIntroduction to machine learning and pattern recognition Lecture 2 Coryn BailerJones
Introduction to machine learning and pattern recognition Lecture 2 Coryn BailerJones http://www.mpia.de/homes/calj/mlpr_mpia2008.html 1 1 Last week... supervised and unsupervised methods need adaptive
More informationMachine Learning, Fall 2012 Homework 2
060 Machine Learning, Fall 202 Homework 2 Instructors: Tom Mitchell, Ziv BarJoseph TA in charge: Selen Uguroglu email: sugurogl@cs.cmu.edu SOLUTIONS Naive Bayes, 20 points Problem. Basic concepts, 0
More informationEEL 851: Biometrics. An Overview of Statistical Pattern Recognition EEL 851 1
EEL 851: Biometrics An Overview of Statistical Pattern Recognition EEL 851 1 Outline Introduction Pattern Feature Noise Example Problem Analysis Segmentation Feature Extraction Classification Design Cycle
More informationBayes Classifiers. CAP5610 Machine Learning Instructor: GuoJun QI
Bayes Classifiers CAP5610 Machine Learning Instructor: GuoJun QI Recap: Joint distributions Joint distribution over Input vector X = (X 1, X 2 ) X 1 =B or B (drinking beer or not) X 2 = H or H (headache
More informationMachine Learning 2017
Machine Learning 2017 Volker Roth Department of Mathematics & Computer Science University of Basel 21st March 2017 Volker Roth (University of Basel) Machine Learning 2017 21st March 2017 1 / 41 Section
More informationLecture 4: Probabilistic Learning
DD2431 Autumn, 2015 1 Maximum Likelihood Methods Maximum A Posteriori Methods Bayesian methods 2 Classification vs Clustering Heuristic Example: Kmeans Expectation Maximization 3 Maximum Likelihood Methods
More informationGenerative classifiers: The Gaussian classifier. Ata Kaban School of Computer Science University of Birmingham
Generative classifiers: The Gaussian classifier Ata Kaban School of Computer Science University of Birmingham Outline We have already seen how Bayes rule can be turned into a classifier In all our examples
More informationL11: Pattern recognition principles
L11: Pattern recognition principles Bayesian decision theory Statistical classifiers Dimensionality reduction Clustering This lecture is partly based on [Huang, Acero and Hon, 2001, ch. 4] Introduction
More informationLecture 12. Talk Announcement. Neural Networks. This Lecture: Advanced Machine Learning. Recap: Generalized Linear Discriminants
Advanced Machine Learning Lecture 2 Neural Networks 24..206 Bastian Leibe RWTH Aachen http://www.vision.rwthaachen.de/ leibe@vision.rwthaachen.de Talk Announcement Yann LeCun (NYU & FaceBook AI) 28..
More informationProbabilistic classification CE717: Machine Learning Sharif University of Technology. M. Soleymani Fall 2016
Probabilistic classification CE717: Machine Learning Sharif University of Technology M. Soleymani Fall 2016 Topics Probabilistic approach Bayes decision theory Generative models Gaussian Bayes classifier
More informationCOM336: Neural Computing
COM336: Neural Computing http://www.dcs.shef.ac.uk/ sjr/com336/ Lecture 2: Density Estimation Steve Renals Department of Computer Science University of Sheffield Sheffield S1 4DP UK email: s.renals@dcs.shef.ac.uk
More informationCS 1955: Machine Learning Problem Set 1
CS 955: Machine Learning Problem Set Douglas Lanman dlanman@brown.edu 7 September Regression Problem Show that the prediction errors y f(x; ŵ) are necessarily uncorrelated with any linear function of
More informationSTA414/2104. Lecture 11: Gaussian Processes. Department of Statistics
STA414/2104 Lecture 11: Gaussian Processes Department of Statistics www.utstat.utoronto.ca Delivered by Mark Ebden with thanks to Russ Salakhutdinov Outline Gaussian Processes Exam review Course evaluations
More information> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 BASEL. Logistic Regression. Pattern Recognition 2016 Sandro Schönborn University of Basel
Logistic Regression Pattern Recognition 2016 Sandro Schönborn University of Basel Two Worlds: Probabilistic & Algorithmic We have seen two conceptual approaches to classification: data class density estimation
More informationExpect Values and Probability Density Functions
Intelligent Systems: Reasoning and Recognition James L. Crowley ESIAG / osig Second Semester 00/0 Lesson 5 8 april 0 Expect Values and Probability Density Functions otation... Bayesian Classification (Reminder...3
More informationProblem Set 2. MAS 622J/1.126J: Pattern Recognition and Analysis. Due: 5:00 p.m. on September 30
Problem Set MAS 6J/1.16J: Pattern Recognition and Analysis Due: 5:00 p.m. on September 30 [Note: All instructions to plot data or write a program should be carried out using Matlab. In order to maintain
More informationDensity Estimation. Seungjin Choi
Density Estimation Seungjin Choi Department of Computer Science and Engineering Pohang University of Science and Technology 77 Cheongamro, Namgu, Pohang 37673, Korea seungjin@postech.ac.kr http://mlg.postech.ac.kr/
More informationUniversity of Cambridge Engineering Part IIB Module 3F3: Signal and Pattern Processing Handout 2:. The Multivariate Gaussian & Decision Boundaries
University of Cambridge Engineering Part IIB Module 3F3: Signal and Pattern Processing Handout :. The Multivariate Gaussian & Decision Boundaries..15.1.5 1 8 6 6 8 1 Mark Gales mjfg@eng.cam.ac.uk Lent
More informationPattern Recognition and Machine Learning
Christopher M. Bishop Pattern Recognition and Machine Learning ÖSpri inger Contents Preface Mathematical notation Contents vii xi xiii 1 Introduction 1 1.1 Example: Polynomial Curve Fitting 4 1.2 Probability
More informationNaïve Bayes classification
Naïve Bayes classification 1 Probability theory Random variable: a variable whose possible values are numerical outcomes of a random phenomenon. Examples: A person s height, the outcome of a coin toss
More informationIntroduction to Statistical Inference
Structural Health Monitoring Using Statistical Pattern Recognition Introduction to Statistical Inference Presented by Charles R. Farrar, Ph.D., P.E. Outline Introduce statistical decision making for Structural
More informationPattern Classification
Pattern Classification All materials in these slides were taken from Pattern Classification (2nd ed) by R. O. Duda, P. E. Hart and D. G. Stork, John Wiley & Sons, 2000 with the permission of the authors
More informationMachine Learning. Gaussian Mixture Models. Zhiyao Duan & Bryan Pardo, Machine Learning: EECS 349 Fall
Machine Learning Gaussian Mixture Models Zhiyao Duan & Bryan Pardo, Machine Learning: EECS 349 Fall 2012 1 The Generative Model POV We think of the data as being generated from some process. We assume
More informationAlgorithmisches Lernen/Machine Learning
Algorithmisches Lernen/Machine Learning Part 1: Stefan Wermter Introduction Connectionist Learning (e.g. Neural Networks) DecisionTrees, Genetic Algorithms Part 2: Norman Hendrich SupportVector Machines
More informationMINIMUM EXPECTED RISK PROBABILITY ESTIMATES FOR NONPARAMETRIC NEIGHBORHOOD CLASSIFIERS. Maya Gupta, Luca Cazzanti, and Santosh Srivastava
MINIMUM EXPECTED RISK PROBABILITY ESTIMATES FOR NONPARAMETRIC NEIGHBORHOOD CLASSIFIERS Maya Gupta, Luca Cazzanti, and Santosh Srivastava University of Washington Dept. of Electrical Engineering Seattle,
More informationAdvanced statistical methods for data analysis Lecture 1
Advanced statistical methods for data analysis Lecture 1 RHUL Physics www.pp.rhul.ac.uk/~cowan Universität Mainz Klausurtagung des GK Eichtheorien exp. Tests... Bullay/Mosel 15 17 September, 2008 1 Outline
More informationMachine Learning 4771
Machine Learning 4771 Instructor: Tony Jebara Topic 11 Maximum Likelihood as Bayesian Inference Maximum A Posteriori Bayesian Gaussian Estimation Why Maximum Likelihood? So far, assumed max (log) likelihood
More informationMachine Learning for Signal Processing Bayes Classification and Regression
Machine Learning for Signal Processing Bayes Classification and Regression Instructor: Bhiksha Raj 11755/18797 1 Recap: KNN A very effective and simple way of performing classification Simple model: For
More informationComputer Vision Group Prof. Daniel Cremers. 2. Regression (cont.)
Prof. Daniel Cremers 2. Regression (cont.) Regression with MLE (Rep.) Assume that y is affected by Gaussian noise : t = f(x, w)+ where Thus, we have p(t x, w, )=N (t; f(x, w), 2 ) 2 Maximum APosteriori
More information9/12/17. Types of learning. Modeling data. Supervised learning: Classification. Supervised learning: Regression. Unsupervised learning: Clustering
Types of learning Modeling data Supervised: we know input and targets Goal is to learn a model that, given input data, accurately predicts target data Unsupervised: we know the input only and want to make
More informationIntelligent Systems Statistical Machine Learning
Intelligent Systems Statistical Machine Learning Carsten Rother, Dmitrij Schlesinger WS2015/2016, Our model and tasks The model: two variables are usually present:  the first one is typically discrete
More informationContents Lecture 4. Lecture 4 Linear Discriminant Analysis. Summary of Lecture 3 (II/II) Summary of Lecture 3 (I/II)
Contents Lecture Lecture Linear Discriminant Analysis Fredrik Lindsten Division of Systems and Control Department of Information Technology Uppsala University Email: fredriklindsten@ituuse Summary of lecture
More informationAnnouncements. Proposals graded
Announcements Proposals graded Kevin Jamieson 2018 1 Bayesian Methods Machine Learning CSE546 Kevin Jamieson University of Washington November 1, 2018 2018 Kevin Jamieson 2 MLE Recap  coin flips Data:
More informationProbabilistic modeling. The slides are closely adapted from Subhransu Maji s slides
Probabilistic modeling The slides are closely adapted from Subhransu Maji s slides Overview So far the models and algorithms you have learned about are relatively disconnected Probabilistic modeling framework
More informationCOMP 551 Applied Machine Learning Lecture 20: Gaussian processes
COMP 55 Applied Machine Learning Lecture 2: Gaussian processes Instructor: Ryan Lowe (ryan.lowe@cs.mcgill.ca) Slides mostly by: (herke.vanhoof@mcgill.ca) Class web page: www.cs.mcgill.ca/~hvanho2/comp55
More informationPattern Recognition. Parameter Estimation of Probability Density Functions
Pattern Recognition Parameter Estimation of Probability Density Functions Classification Problem (Review) The classification problem is to assign an arbitrary feature vector x F to one of c classes. The
More informationIntroduction to Graphical Models
Introduction to Graphical Models The 15 th Winter School of Statistical Physics POSCO International Center & POSTECH, Pohang 2018. 1. 9 (Tue.) YungKyun Noh GENERALIZATION FOR PREDICTION 2 Probabilistic
More informationLecture 4 Discriminant Analysis, knearest Neighbors
Lecture 4 Discriminant Analysis, knearest Neighbors Fredrik Lindsten Division of Systems and Control Department of Information Technology Uppsala University. Email: fredrik.lindsten@it.uu.se fredrik.lindsten@it.uu.se
More informationStatistical Methods for Particle Physics Lecture 1: parameter estimation, statistical tests
Statistical Methods for Particle Physics Lecture 1: parameter estimation, statistical tests http://benasque.org/2018tae/cgibin/talks/allprint.pl TAE 2018 Benasque, Spain 315 Sept 2018 Glen Cowan Physics
More informationMachine learning for pervasive systems Classification in highdimensional spaces
Machine learning for pervasive systems Classification in highdimensional spaces Department of Communications and Networking Aalto University, School of Electrical Engineering stephan.sigg@aalto.fi Version
More informationStatistical and Learning Techniques in Computer Vision Lecture 1: Random Variables Jens Rittscher and Chuck Stewart
Statistical and Learning Techniques in Computer Vision Lecture 1: Random Variables Jens Rittscher and Chuck Stewart 1 Motivation Imaging is a stochastic process: If we take all the different sources of
More information10701/ Machine Learning  Midterm Exam, Fall 2010
10701/15781 Machine Learning  Midterm Exam, Fall 2010 Aarti Singh Carnegie Mellon University 1. Personal info: Name: Andrew account: Email address: 2. There should be 15 numbered pages in this exam
More informationBayesian Decision Theory
Bayesian Decision Theory Selim Aksoy Department of Computer Engineering Bilkent University saksoy@cs.bilkent.edu.tr CS 551, Fall 2017 CS 551, Fall 2017 c 2017, Selim Aksoy (Bilkent University) 1 / 46 Bayesian
More informationCMUQ Lecture 24:
CMUQ 15381 Lecture 24: Supervised Learning 2 Teacher: Gianni A. Di Caro SUPERVISED LEARNING Hypotheses space Hypothesis function Labeled Given Errors Performance criteria Given a collection of input
More informationINTRODUCTION TO PATTERN RECOGNITION
INTRODUCTION TO PATTERN RECOGNITION INSTRUCTOR: WEI DING 1 Pattern Recognition Automatic discovery of regularities in data through the use of computer algorithms With the use of these regularities to take
More informationCS534 Machine Learning  Spring Final Exam
CS534 Machine Learning  Spring 2013 Final Exam Name: You have 110 minutes. There are 6 questions (8 pages including cover page). If you get stuck on one question, move on to others and come back to the
More informationNaïve Bayes classification. p ij 11/15/16. Probability theory. Probability theory. Probability theory. X P (X = x i )=1 i. Marginal Probability
Probability theory Naïve Bayes classification Random variable: a variable whose possible values are numerical outcomes of a random phenomenon. s: A person s height, the outcome of a coin toss Distinguish
More informationCSCI567: Machine Learning (Spring 2019)
CSCI567: Machine Learning (Spring 2019) Prof. Victor Adamchik U of Southern California Mar. 19, 2019 March 19, 2019 1 / 43 Administration March 19, 2019 2 / 43 Administration TA3 is due this week March
More informationMachine Learning Linear Classification. Prof. Matteo Matteucci
Machine Learning Linear Classification Prof. Matteo Matteucci Recall from the first lecture 2 X R p Regression Y R Continuous Output X R p Y {Ω 0, Ω 1,, Ω K } Classification Discrete Output X R p Y (X)
More informationCheng Soon Ong & Christian Walder. Canberra February June 2018
Cheng Soon Ong & Christian Walder Research Group and College of Engineering and Computer Science Canberra February June 218 Outlines Overview Introduction Linear Algebra Probability Linear Regression 1
More informationCOMP90051 Statistical Machine Learning
COMP90051 Statistical Machine Learning Semester 2, 2017 Lecturer: Trevor Cohn 2. Statistical Schools Adapted from slides by Ben Rubinstein Statistical Schools of Thought Remainder of lecture is to provide
More informationLecture 3: Pattern Classification
EE E6820: Speech & Audio Processing & Recognition Lecture 3: Pattern Classification 1 2 3 4 5 The problem of classification Linear and nonlinear classifiers Probabilistic classification Gaussians, mixtures
More informationIntroduction to Machine Learning
Outline Introduction to Machine Learning Bayesian Classification Varun Chandola March 8, 017 1. {circular,large,light,smooth,thick}, malignant. {circular,large,light,irregular,thick}, malignant 3. {oval,large,dark,smooth,thin},
More informationECE521 Lecture7. Logistic Regression
ECE521 Lecture7 Logistic Regression Outline Review of decision theory Logistic regression A single neuron Multiclass classification 2 Outline Decision theory is conceptually easy and computationally hard
More informationProbability and Information Theory. Sargur N. Srihari
Probability and Information Theory Sargur N. srihari@cedar.buffalo.edu 1 Topics in Probability and Information Theory Overview 1. Why Probability? 2. Random Variables 3. Probability Distributions 4. Marginal
More information