Machine Learning for Signal Processing Bayes Classification and Regression


 Merryl Lynch
 1 years ago
 Views:
Transcription
1 Machine Learning for Signal Processing Bayes Classification and Regression Instructor: Bhiksha Raj 11755/
2 Recap: KNN A very effective and simple way of performing classification Simple model: For any instance, select the class from the instances close to it in feature space 11755/
3 Multiclass Image Classification 3
4 knearest Neighbors Given a query item: Find k closest matches in a labeled dataset 4
5 knearest Neighbors Given a query item: Find k closest matches Return the most Frequent label 5
6 knearest Neighbors 2 votes for cat, 1 each for Buffalo, Cat wins Deer, Lion 7
7 Nearest neighbor method Weighted majority vote within the k nearest neighbors Y x = argmax c x i Nk x,y i =c w(x, x i )y i new K= 1: blue K= 3: green 8
8 But what happens if.. You have many training instances at exactly that value of x? Majority vote on nearest neighbors: new Test Training y = argmax y count(y x) 9
9 But what happens if.. You have many training instances at exactly that value of x? Majority vote on nearest neighbors: new Test Training y = argmax y N y (x) N(x) 10
10 But what happens if.. You have many training instances at exactly that value of x? Majority vote on nearest neighbors: new Test Training y = argmax y P(y x) 11
11 But what happens if.. You have many training instances at exactly that value of x? Majority vote on nearest neighbors: new Test Training y = argmax y P(y x) Bayes Classification Rule 12
12 Bayes Classification Rule For any observed feature X, select the class value Y that is most frequent Also applies to continuous valued predicted variables I.e. regression Select Y to maximize the a posteriori probability P(Y X) Bayes classification is an instance of maximum a posteriori estimation 11755/
13 Bayes classification What happens if there are no exact neighbors No training instances with exactly the same X value? new 14
14 Given Bayes Classification Rule a set of classes C = {C 1, C 2,, C N } Conditional probability distributions P(C X) Classification performed as መC = argmax C C P(C X) Problem: How do you characterize P C X Require a function that, given an X, computes P(C X) for every class C 11755/
15 Modelling P(C X) Each pixel is a combination of red green and blue weighted by the a posteriori probability of the classes Blue: Class 1 Red: Class 2 Green: Class 3 Assumption: there s a continuous function that, at every X, produces a vector of outputs P(C X) for every class C The decision boundary for any class is the boundary within which its own posterior has the highest value This function accurately represents the actual a posteriori probabilities for the classes Objective: Estimate this function 11755/
16 Modelling the posterior To model the posterior, we need a functional form for P(C X) which can be learned Typically this functional form is expressed in terms of distance from a decision boundary The simplest decision boundary is the linear boundary 11755/
17 Bayesian Linear Classification: Two class case First: Twoclass classification Assumption: the decision boundary between the classes is a simple hyperplane As you go away from the hyperplane, the fraction of data from one class increases, while that from the other decreases Will also hold for any sample of data 11755/
18 1D binary class example y x Onedimensional example for visualization Only two classes (represented by y=0 and y=1) All (red) dots at Y=1 represent instances of class Y=1 All (blue) dots at Y=0 are from class Y=0 The data are not linearly separable In this 1D example, a linear separator is a threshold No threshold will cleanly separate red and blue dots 19
19 The probability of y=1 y x Consider this differently: at each point look at a small window around that point Plot the average value within the window This is an approximation of the probability of Y=1 at that point 20
20 The probability of y=1 y x Consider this differently: at each point look at a small window around that point Plot the average value within the window This is an approximation of the probability of 1 at that point 21
21 The probability of y=1 y x Consider this differently: at each point look at a small window around that point Plot the average value within the window This is an approximation of the probability of 1 at that point 22
22 The probability of y=1 y x Consider this differently: at each point look at a small window around that point Plot the average value within the window This is an approximation of the probability of 1 at that point 23
23 The probability of y=1 y x Consider this differently: at each point look at a small window around that point Plot the average value within the window This is an approximation of the probability of 1 at that point 24
24 The probability of y=1 y x Consider this differently: at each point look at a small window around that point Plot the average value within the window This is an approximation of the probability of 1 at that point 25
25 The probability of y=1 y x Consider this differently: at each point look at a small window around that point Plot the average value within the window This is an approximation of the probability of 1 at that point 26
26 The probability of y=1 y x Consider this differently: at each point look at a small window around that point Plot the average value within the window This is an approximation of the probability of 1 at that point 27
27 The probability of y=1 y x Consider this differently: at each point look at a small window around that point Plot the average value within the window This is an approximation of the probability of 1 at that point 28
28 The probability of y=1 y x Consider this differently: at each point look at a small window around that point Plot the average value within the window This is an approximation of the probability of 1 at that point 29
29 The probability of y=1 y x Consider this differently: at each point look at a small window around that point Plot the average value within the window This is an approximation of the probability of 1 at that point 30
30 The probability of y=1 y x Consider this differently: at each point look at a small window around that point Plot the average value within the window This is an approximation of the probability of 1 at that point 31
31 The probability of y=1 y x Consider this differently: at each point look at a small window around that point Plot the average value within the window This is an approximation of the probability of 1 at that point 32
32 The probability of y=1 Need a function to model the shape of this curve. y A good choice that works under many conditions: Logistic function x Consider this differently: at each point look at a small window around that point Plot the average value within the window This is an approximation of the probability of 1 at that point 33
33 The logistic regression model y=1 P ( y x) = w ( 0 + x w 1+ e 1 ) y=0 x Class 1 becomes increasingly probable going left to right Very typical in many problems The logistic is a function of the distance from the P(y x) = 0.5 boundary 34
34 For twodimensional input Decision: y > 0.5? x 2 1 When X is a 2D variable P( y x) = w ( 0 + x ) 1+ e w The decision boundary for P(Y X)=0.5 is a hyperplane It is a linear model x 1 35
35 The logistic regression model y=1 P e w0 ( y x) = w 0 + w x = w ( 0 + w x 1+ e + w x 1+ e 1 ) y=0 x Note how it varies with w 0 + w 1 x 36
36 Estimating the model y P ( y x) = f ( x) = ( w w x) 1+ e 1 + x Given the training data (many (x, y) pairs represented by the dots), estimate w 0 and w 1 for the curve 37
37 Estimating the model 38 x y ) ( 1 1 ) 1 ( x w w e x y P + + = = ) ( 1 1 ) 1 ( x w w e x y P + + = = ) ( 1 1 ) ( x w w y e x y P + + = Easier to represent using a y = +1/1 notation
38 Estimating the model Given: Training data X 1, y 1, X 2, y 2,, X N, y N Xs are vectors, ys are binary (0/1) class values Total probability of data P X 1, y 1, X 2, y 2,, X N, y N = P X i, y i 1 = P y i X i P X i = 1 + e P X y i(w 0 +w T X i ) i i i i 39
39 Estimating the model Likelihood P Training data = Log likelihood log P Training data = i e y i(w 0 +w T X i ) P X i log P X i log 1 + e y i(w 0 +w T X i ) i i 40
40 Maximum Likelihood Estimate w 0, w 1 = argmax w 0,w 1 log P Training data Equals (note argmin rather than argmax) w 0, w 1 = argmin w 0,w i log 1 + e y i(w 0 +w T X i ) Minimizing the KL divergence between the desired output y and actual output 1 1+e (w 0+w T X i ) Cannot be solved directly, needs gradient descent 41
41 Model learned by logistic regression Pure Red: 0 Pure Green: 1 The figure shows the class probability over a twodimensional feature space Any decision threshold P(C X) = Const is a hyperplane Diagonal line in this case 11755/
42 Multiclass logistic regression The simple logistic regression model can be extended to multiple classes: P C X = exp(w C T X) σ C exp(w T C X) W T C X is, in fact, the discriminant function of the classes We ve encountered discriminant functions earlier Also called a softmax Each class C i has a probability that is exponentially related to the closeness of the vector X to a representative vector w i for the class This too can be learned from training data via maximum likelihood estimation Just like the twoclass case 11755/
43 Multiclass logistic regression The boundary between adjacent classes is a hyperplane (line) The decision boundary for any class is convex polytope with hyperplane segments I.e. still a linear classifier 11755/
44 Multiclass Bayes In many classification problems, linear boundaries are not sufficient We need to be able to model more complex boundaries This too can be supported by the logistic regression classifier 11755/
45 Logistic regression with nonlinear boundaries The logistic regression can be modified to have nonlinear discriminants: P C X = exp(f(x; θ C)) σ C exp(f(x; θ C )) f(x; θ C ) is the discriminant for class C, and has parameter θ C The discriminants determine the shape of the decision boundary Nonlinear discriminants result in nonlinear decision boundaries The parameters θ C for all classes can be learned by maximum likelihood (or MAP) estimation as before 11755/
46 Quadratic discriminant P C X = exp(f(x; θ C)) σ C exp(f(x; θ C )) With quadratic discriminants: f X; θ C = (X α C )β C (X α C ) T Note that decision boundaries are quadratic The probability of a class increases (or decreases) as we go away from a boundary 11755/
47 Logistic regression with nonlinear boundaries P C X = exp(f(x; θ C)) σ C exp(f(x; θ C )) For complex decision boundaries, the function f(x; θ C ) must be correspondingly complex Currently the most successful approach in these cases is to model f(x; θ C ) by a neural network In fact neural networks with softmax decision layers may be seen as an instance of a logistic regression with a nonlinear discriminant Topic for a later class 11755/
48 Logistic regression with nonlinear boundaries The logistic regression can be modified to have nonlinear discriminants: P C X = exp(f(x; θ C)) σ C exp(f(x; θ C )) Note: This can also be viewed as nonlinearly transforming the data X into a space where a simple linear logistic regression models posteriors well Z X = [f X; θ 1 f X; θ 2 f(x; θ K )] T I.e. into a space where the data are most linearly separable We will discuss this in a later lecture on neural networks 11755/
49 Problem with modelling P(C X) We have considered modelling the a posteriori probability of the classes directly This implicitly assumes that The characteristics of the data for any class remain the same between train and test The relative proportions of the classes too remain the same Often the second assumption will not hold The data characteristics remain, but the relative proportions change E.g. the shapes of the differently colored coins don t change, but the relative proportions of the colors changes between train and test We must then modify our approach to Bayes classification to a generative framework 11755/
50 The Bayesian Classifier.. መC = argmax P(C X) C C Choose the class that is most frequent for the given X P(C X) = P(C)P(X C) P(X) argmax C C P(C X) = argmax C C P(C)P(X C) Choose the class that is most likely to have produced X While accounting for the relative frequency of C 11755/
51 P(X) Bayes Classification Rule Given a set of classes C = {C 1, C 2,, C N } መC = argmax P(C)P(X C) C C P(X C i ) measures the probability that a random instance of class C i will take the value X P(X C 1 ) X 11755/
52 P(X) Bayes Classification Rule Given a set of classes C = {C 1, C 2,, C N } መC = argmax P(C)P(X C) C C P(X C i ) measures the probability that a random instance of class C i will take the value X P(X C 2 ) P(X C 1 ) X 11755/
53 P(X) Bayes Classification Rule Given a set of classes C = {C 1, C 2,, C N } መC = argmax C C P(C)P(X C) P(C i ) scales them up to match the expected relative proportions of the classes P(C 2 )P(X C 2 ) P(C 1 )P(X C 1 ) X 11755/
54 P(X) Bayes Classification Rule Given a set of classes C = {C 1, C 2,, C N } መC = argmax C C P(C)P(X C) P(C i ) scales them up to match the expected relative proportions of the classes Decision boundary P(C 2 )P(X C 2 ) P(C 1 )P(X C 1 ) X 11755/
55 P(X) Bayes Classification Rule Given a set of classes C = {C 1, C 2,, C N } መC = argmax C C P(C)P(X C) P(C i ) scales them up to match the expected relative proportions of the classes Fraction of all instances that belong to C 1 and fall on the wrong Decision boundary side of the boundary and are misclassified P(C 1 )P(X C 1 ) P(C 2 )P(X C 2 ) X 11755/
56 P(X) Bayes Classification Rule Given a set of classes C = {C 1, C 2,, C N } መC = argmax C C P(C)P(X C) P(C i ) scales them up to match the expected relative proportions of the classes Fraction of all instances that belong to C 2 and fall on the wrong Decision boundary side of the boundary and are misclassified P(C 1 )P(X C 1 ) P(C 2 )P(X C 2 ) X 11755/
57 P(X) Bayes Classification Rule Given a set of classes C = {C 1, C 2,, C N } መC = argmax C C P(C)P(X C) P(C i ) scales them up to match the expected relative proportions of the classes Decision boundary Total misclassification probability P(C 2 )P(X C 2 ) P(C 1 )P(X C 1 ) X 11755/
58 P(X) Bayes Classification Rule The Bayes classification rule is the statistically optimal classification rule Moving the boundary in either direction will always increase the classification error Excess classification error with shifted boundary P(C 2 )P(X C 2 ) P(C 1 )P(X C 1 ) X 11755/
59 The Bayesian Classifier.. መC = argmax C C P(C X) = argmax C C P(C)P(X C) We can now directly learn the classspecific statistical characteristics P(X C) from the training data The relative frequency of C, P(C), can be independently adjusted to our expectations of these frequencies in the test data These need not match the training data 11755/
60 Modeling P(X C) Challenge: How to learn P(X C) This will not be known beforehand and must be learned from examples of X that belong to class C Will generally have unknown and unknowable shape We only observe samples of X Must make some assumptions about the form of P(X C) 11755/
61 The problem of dependent variables P(X C) = P(X 1, X 2,, X D C) must be defined for every combination of X 1, X 2,, X D Too many parameters to describe explicitly Most combinations unseen in training data P(X C) may have an arbitrary scatter/shape Hard to characterize mathematically Typically do so by assigning a functional form to it 11755/
62 The problem of dependent variables P(X C) = P(X 1, X 2,, X D C) must be defined for every combination of X 1, X 2,, X D Too many parameters to describe explicitly Most combinations unseen in training data P(X C) may have an arbitrary scatter/shape Hard to characterize mathematically Typically do so by assigning a functional form to it 11755/
63 The Naïve Bayes assumption C C X 1 X 5 X2 X 4 X 1 X 2 X 3 X 4 X 5 X 3 Assume all the components are independent of one another The joint probability is the product of the marginal P X C = P X 1, X 2,, X D C = P X i C Sufficient to learn marginal distributions P X i C The problem of having to observe all combinations of X 1, X 2,, X D never arises i 11755/
64 Naïve Bayes estimating P X i C P X i C may be estimated using conventional maximum likelihood estimation Given a number of training instances belonging to class C Select the ith component of all instances Estimate P X i C For discretevalued X i this will be a multinomial distribution For continuous valued X i a form must be assumed E.g Gaussian, Laplacian etc 11755/
65 Naïve Bayes Binary Case 11755/
66 The problem of dependent variables P(X C) = P(X 1, X 2,, X D C) must be defined for every combination of X 1, X 2,, X D Too many parameters Most combinations unseen in training data P(X C) may have an arbitrary scatter/shape Hard to characterize mathematically Typically do so by assigning a functional form to it 11755/
67 Assigning a functional form to P(X C) Assign a functional form to P(X C) Common assumptions: Unimodal forms: Gaussian, Laplacian Multimodal forms: Gaussian mixtures Time series: Hidden Markov models Multidimensional structures: Markov random fields 11755/
68 Assigning a functional form to P(X C) Assign a functional form to P(X C) Common assumptions: Unimodal forms: Gaussian, Laplacian Most common of all Multimodal forms: Gaussian mixtures Time series: Hidden Markov models Multidimensional structures: Markov random fields 11755/
69 Gaussian Distribution Mean Vector Covariance Matrix  Symmetric  Positive Definite 11755/
70 Gaussian Distribution 11755/
71 Parameter Estimation 11755/
72 Gaussian classifier Different Classes, different Gaussians 11755/
73 Gaussian Classifier For each class we need: Mean Vector Covariance Matrix Training Fit a Gaussian to each class Classification: Problem: Find the best Gaussian to explain the distribution for the class arg max i Many parameters to train! P C i p x, μ i, Σ i Dominated by covariance: for Ddimensional data the covariance matrices requires D 2 parameters each For N c classes, a total of N c D 2 parameters 11755/
74 Homoskedasticity assumption Assume all distributions have the same covariance Σ i = Σ i Assumption, may not be true But still works in many cases Fewer parameters to train One common covariance matrix for all classes Only D 2 total parameters As opposed to N c D 2 if each class has its own covariance matrix 11755/
75 Homoskedastic Gaussians Σ 1 = Σ 2 = = Σ K = Σ For the binary classification case (K = 2) Decision boundary: P X C 2 P C 2 = P X C 2 P C /
76 Homoskedastic Gaussians Linear Boundary!!! 11755/
77 Homoskedastic Gaussians 11755/
78 Homoskedastic Gaussians, K > 2 Case K > 2 (more than two classes) Classification performed as: argmax P C i p x, μ i, Σ i i Taking logs and ignoring the common constant argmax i 1 2 x μ i T Σ 1 x μ i + log P C i Expanding out and ignoring common terms argmax i This is just a linear classifier 1 2 xt Σ 1 μ i 1 2 μ i T Σ 1 μ i + log P C i 11755/
79 Homoskedastic Gaussians, K > 2 Decision boundaries for argmax P C i p x, μ i, Σ i i Linear classifier: Decision boundaries are hyperplanes 11755/
80 Homoskedastic Gaussians, K > 2 Case K > 2 (more than two classes) Classification performed as: argmax P C i p x, μ i, Σ i i Taking logs and ignoring the common constant argmax i 1 2 x μ i T Σ 1 x μ i + log P C i Changing the sign and rewriting it as argmin argmin i x μ i T Σ 1 x μ i 2log P C i 11755/
81 Homoskedastic Gaussians Mahalanobis Distance A Gaussian Classifier with common Covariance Matrix is similar to a Nearest Neighbor Classifier Classification corresponds to the nearest mean vector 11755/
82 How to estimate the Covariance Matrix? Maximum likelihood estimate of covariances of individual classes: Σ C = 1 N C N C x i (c) μc x i (c) μc T i=1 Estimate of common covariance for all classes Σ = 1 σk C =1 K N C Σ C N C C= /
83 Hetero skedastic Gaussians.. Homoskedastic Gaussians do not capture nonlinear decision boundaries Also, the assumption that all Gaussians have the same covariance is questionable Permitting each Gaussian to have its own covariance results in nonlinear decision boundaries Hetero skedastic Gaussians 11755/
84 Heteroskedastic Gaussians Different Covariance Matrices 1D case. K = 2 Decision Boundary Quadratic Boundary!!! 11755/
85 Heteroskedastic Gaussians 11755/
86 Digit recognition Handwriting recognition Money detection in check deposit $ /
87 now is to take a probabilistic view of the problem: We treat the variations in the d Gaussian distribution Classifier over the images. So for each for digit j 2 Digit { 0,1,...,9}, we have a probabil ion of the distribution P 1 (x) that captures our uncertainty of the variations among digit 1. over the images x, which represents our uncertainty or imperfect knowledge about the P j are imperfect models of the digits. Nevertheless, images containing since they are digit concentrated j. around ch digit, they are still a useful representation of our recognition knowledge. At this point is still not t the distributions P j from, but for the moment Under let usdistribution assume we have P j (x), them. we are a lot more likely to generate images x that look like digit j; however, there is also a small probability that P j (x) generates images that are e can phrase the digit classification problem in terms of balls and bins as follows: We have digit j, but by random chance may look like other digits. Figure 3 shows an illustrati corresponds to a digit. Within each bin we have all possible balls (images) corresponding Weassume the images within bin j are distributed P 1 (x). according The bulk to the of the probability mass (images distribution that are assigned high probability) are good image he images in bin j are more likely to have a digit the tails j written (images in them. withthen small weprobability) are given an are some ambiguous images that could be int taken from one of the bins (the image has apossibly digit written other in it), digits. but we don t know which is to guess which bin the image was likely to have come from (which digit the image s and bins analogy of digit classification. Balls Figure are the3: images, Illustration and bins of are the the distribution digits. P 1 (x) that captures our uncertainty of the vari The distributions P j are imperfect models of the digits. Nevertheless, since they are good images of each digit, they are still a useful representation of our knowledge. At t clear where we get the distributions P j from, but for the moment let us assume we hav ecture 21 3 With this setup, we can phrase the digit classification problem in terms of balls and bin 10 bins, each one corresponds to a digit. Within each bin we have all possible balls (im to that bin (digit). Weassume the images within bin j aredistributed according to the pr P j, which means / the images in bin j are more likely to have a digit j written 88 in them.
88 Showing the average of each digit Average digit Figure 7: Mean vectors for each digit (the estimated p ji s) /
89 Bayes classifcation Back to balls and bins Suppose we are just given a ball (image) x Which bin (digit) y did it come from? Normalize the Posterior prior posterior Bayes rule /
90 Inadequacy of Gaussian classifiers Gaussian classifiers can only capture simple linear or quadratic decision boundaries Often, the decision boundaries required are more complex In this case we must employ a Gaussian Mixture classifier argmax P C i p x C i i p x C i is modelled by a Gaussian mixture 11755/
91 GMM classifier  For each class, train a GMM (with EM)  Classify according to 11755/
92 Bayesian Classification with Gaussian Mixtures Plotting P C i p x C i for all classes Left: Twoclass classification, Right: Threeclass classification Each class modelled by a mixture of three Gaussians Note the complex decision boundary 11755/
93 Estimating P(C) argmax P C i p x, μ i, Σ i i Have not explained where the class prior P C i from comes This can be dependent on the test data Typical solutions: Estimate from training data Optimize on development or heldout test data Heuristic guess Conservative estimates Set the prior of classes that have high cost if incorrectly detected to be low Set prior of classes that have high cost if incorrectly missed to be low Etc /
94 Topics not covered Maximum a posteriori estimation When we make assumptions about the parameters (means, covariances) themselves MAP regression with Gaussians Predicting continuousvalued RVs assuming Gaussian distributions MAP regression with Gaussian Mixtures Predicting continuousvalued RVs with Gaussian mixture distributions Timeseries and other structured data Partially covered in a later lecture 11755/
Machine Learning for Signal Processing Bayes Classification
Machine Learning for Signal Processing Bayes Classification Class 16. 24 Oct 2017 Instructor: Bhiksha Raj  Abelino Jimenez 11755/18797 1 Recap: KNN A very effective and simple way of performing classification
More informationMachine Learning Lecture 5
Machine Learning Lecture 5 Linear Discriminant Functions 26.10.2017 Bastian Leibe RWTH Aachen http://www.vision.rwthaachen.de leibe@vision.rwthaachen.de Course Outline Fundamentals Bayes Decision Theory
More informationDiscrete Mathematics and Probability Theory Fall 2015 Lecture 21
CS 70 Discrete Mathematics and Probability Theory Fall 205 Lecture 2 Inference In this note we revisit the problem of inference: Given some data or observations from the world, what can we infer about
More informationClassification CE717: Machine Learning Sharif University of Technology. M. Soleymani Fall 2012
Classification CE717: Machine Learning Sharif University of Technology M. Soleymani Fall 2012 Topics Discriminant functions Logistic regression Perceptron Generative models Generative vs. discriminative
More informationCSCI567: Machine Learning (Spring 2019)
CSCI567: Machine Learning (Spring 2019) Prof. Victor Adamchik U of Southern California Mar. 19, 2019 March 19, 2019 1 / 43 Administration March 19, 2019 2 / 43 Administration TA3 is due this week March
More informationSTA 4273H: Statistical Machine Learning
STA 4273H: Statistical Machine Learning Russ Salakhutdinov Department of Statistics! rsalakhu@utstat.toronto.edu! http://www.utstat.utoronto.ca/~rsalakhu/ Sidney Smith Hall, Room 6002 Lecture 3 Linear
More informationLinear Models for Classification
Linear Models for Classification Oliver Schulte  CMPT 726 Bishop PRML Ch. 4 Classification: Handwritten Digit Recognition CHINE INTELLIGENCE, VOL. 24, NO. 24, APRIL 2002 x i = t i = (0, 0, 0, 1, 0, 0,
More informationNaïve Bayes classification
Naïve Bayes classification 1 Probability theory Random variable: a variable whose possible values are numerical outcomes of a random phenomenon. Examples: A person s height, the outcome of a coin toss
More informationCh 4. Linear Models for Classification
Ch 4. Linear Models for Classification Pattern Recognition and Machine Learning, C. M. Bishop, 2006. Department of Computer Science and Engineering Pohang University of Science and echnology 77 Cheongamro,
More informationProbabilistic classification CE717: Machine Learning Sharif University of Technology. M. Soleymani Fall 2016
Probabilistic classification CE717: Machine Learning Sharif University of Technology M. Soleymani Fall 2016 Topics Probabilistic approach Bayes decision theory Generative models Gaussian Bayes classifier
More informationLecture 3: Pattern Classification
EE E6820: Speech & Audio Processing & Recognition Lecture 3: Pattern Classification 1 2 3 4 5 The problem of classification Linear and nonlinear classifiers Probabilistic classification Gaussians, mixtures
More informationBayesian Learning (II)
Universität Potsdam Institut für Informatik Lehrstuhl Maschinelles Lernen Bayesian Learning (II) Niels Landwehr Overview Probabilities, expected values, variance Basic concepts of Bayesian learning MAP
More informationStatistical Data Mining and Machine Learning Hilary Term 2016
Statistical Data Mining and Machine Learning Hilary Term 2016 Dino Sejdinovic Department of Statistics Oxford Slides and other materials available at: http://www.stats.ox.ac.uk/~sejdinov/sdmml Naïve Bayes
More informationNaïve Bayes classification. p ij 11/15/16. Probability theory. Probability theory. Probability theory. X P (X = x i )=1 i. Marginal Probability
Probability theory Naïve Bayes classification Random variable: a variable whose possible values are numerical outcomes of a random phenomenon. s: A person s height, the outcome of a coin toss Distinguish
More informationDay 5: Generative models, structured classification
Day 5: Generative models, structured classification Introduction to Machine Learning Summer School June 18, 2018  June 29, 2018, Chicago Instructor: Suriya Gunasekar, TTI Chicago 22 June 2018 Linear regression
More informationBayes Classifiers. CAP5610 Machine Learning Instructor: GuoJun QI
Bayes Classifiers CAP5610 Machine Learning Instructor: GuoJun QI Recap: Joint distributions Joint distribution over Input vector X = (X 1, X 2 ) X 1 =B or B (drinking beer or not) X 2 = H or H (headache
More informationGaussian and Linear Discriminant Analysis; Multiclass Classification
Gaussian and Linear Discriminant Analysis; Multiclass Classification Professor Ameet Talwalkar Slide Credit: Professor Fei Sha Professor Ameet Talwalkar CS260 Machine Learning Algorithms October 13, 2015
More informationProbabilistic modeling. The slides are closely adapted from Subhransu Maji s slides
Probabilistic modeling The slides are closely adapted from Subhransu Maji s slides Overview So far the models and algorithms you have learned about are relatively disconnected Probabilistic modeling framework
More informationMachine Learning Basics Lecture 7: Multiclass Classification. Princeton University COS 495 Instructor: Yingyu Liang
Machine Learning Basics Lecture 7: Multiclass Classification Princeton University COS 495 Instructor: Yingyu Liang Example: image classification indoor Indoor outdoor Example: image classification (multiclass)
More informationThe Bayes classifier
The Bayes classifier Consider where is a random vector in is a random variable (depending on ) Let be a classifier with probability of error/risk given by The Bayes classifier (denoted ) is the optimal
More informationNotes on Discriminant Functions and Optimal Classification
Notes on Discriminant Functions and Optimal Classification Padhraic Smyth, Department of Computer Science University of California, Irvine c 2017 1 Discriminant Functions Consider a classification problem
More information9/12/17. Types of learning. Modeling data. Supervised learning: Classification. Supervised learning: Regression. Unsupervised learning: Clustering
Types of learning Modeling data Supervised: we know input and targets Goal is to learn a model that, given input data, accurately predicts target data Unsupervised: we know the input only and want to make
More informationMidterm Review CS 6375: Machine Learning. Vibhav Gogate The University of Texas at Dallas
Midterm Review CS 6375: Machine Learning Vibhav Gogate The University of Texas at Dallas Machine Learning Supervised Learning Unsupervised Learning Reinforcement Learning Parametric Y Continuous Nonparametric
More information18.9 SUPPORT VECTOR MACHINES
744 Chapter 8. Learning from Examples is the fact that each regression problem will be easier to solve, because it involves only the examples with nonzero weight the examples whose kernels overlap the
More informationMark your answers ON THE EXAM ITSELF. If you are not sure of your answer you may wish to provide a brief explanation.
CS 189 Spring 2015 Introduction to Machine Learning Midterm You have 80 minutes for the exam. The exam is closed book, closed notes except your onepage crib sheet. No calculators or electronic items.
More informationLinear & nonlinear classifiers
Linear & nonlinear classifiers Machine Learning Hamid Beigy Sharif University of Technology Fall 1394 Hamid Beigy (Sharif University of Technology) Linear & nonlinear classifiers Fall 1394 1 / 34 Table
More information> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 BASEL. Logistic Regression. Pattern Recognition 2016 Sandro Schönborn University of Basel
Logistic Regression Pattern Recognition 2016 Sandro Schönborn University of Basel Two Worlds: Probabilistic & Algorithmic We have seen two conceptual approaches to classification: data class density estimation
More informationCMUQ Lecture 24:
CMUQ 15381 Lecture 24: Supervised Learning 2 Teacher: Gianni A. Di Caro SUPERVISED LEARNING Hypotheses space Hypothesis function Labeled Given Errors Performance criteria Given a collection of input
More informationIntroduction to Machine Learning
Introduction to Machine Learning Bayesian Classification Varun Chandola Computer Science & Engineering State University of New York at Buffalo Buffalo, NY, USA chandola@buffalo.edu Chandola@UB CSE 474/574
More informationMachine Learning Practice Page 2 of 2 10/28/13
Machine Learning 10701 Practice Page 2 of 2 10/28/13 1. True or False Please give an explanation for your answer, this is worth 1 pt/question. (a) (2 points) No classifier can do better than a naive Bayes
More informationMidterm Review CS 7301: Advanced Machine Learning. Vibhav Gogate The University of Texas at Dallas
Midterm Review CS 7301: Advanced Machine Learning Vibhav Gogate The University of Texas at Dallas Supervised Learning Issues in supervised learning What makes learning hard Point Estimation: MLE vs Bayesian
More informationPATTERN RECOGNITION AND MACHINE LEARNING CHAPTER 2: PROBABILITY DISTRIBUTIONS
PATTERN RECOGNITION AND MACHINE LEARNING CHAPTER 2: PROBABILITY DISTRIBUTIONS Parametric Distributions Basic building blocks: Need to determine given Representation: or? Recall Curve Fitting Binary Variables
More informationMachine learning comes from Bayesian decision theory in statistics. There we want to minimize the expected value of the loss function.
Bayesian learning: Machine learning comes from Bayesian decision theory in statistics. There we want to minimize the expected value of the loss function. Let y be the true label and y be the predicted
More informationIntroduction to Machine Learning. Maximum Likelihood and Bayesian Inference. Lecturers: Eran Halperin, Lior Wolf
1 Introduction to Machine Learning Maximum Likelihood and Bayesian Inference Lecturers: Eran Halperin, Lior Wolf 201415 We know that X ~ B(n,p), but we do not know p. We get a random sample from X, a
More informationCOM336: Neural Computing
COM336: Neural Computing http://www.dcs.shef.ac.uk/ sjr/com336/ Lecture 2: Density Estimation Steve Renals Department of Computer Science University of Sheffield Sheffield S1 4DP UK email: s.renals@dcs.shef.ac.uk
More informationLogistic Regression. Machine Learning Fall 2018
Logistic Regression Machine Learning Fall 2018 1 Where are e? We have seen the folloing ideas Linear models Learning as loss minimization Bayesian learning criteria (MAP and MLE estimation) The Naïve Bayes
More informationMachine Learning 2017
Machine Learning 2017 Volker Roth Department of Mathematics & Computer Science University of Basel 21st March 2017 Volker Roth (University of Basel) Machine Learning 2017 21st March 2017 1 / 41 Section
More informationLINEAR CLASSIFICATION, PERCEPTRON, LOGISTIC REGRESSION, SVC, NAÏVE BAYES. Supervised Learning
LINEAR CLASSIFICATION, PERCEPTRON, LOGISTIC REGRESSION, SVC, NAÏVE BAYES Supervised Learning Linear vs non linear classifiers In KNN we saw an example of a nonlinear classifier: the decision boundary
More informationCheng Soon Ong & Christian Walder. Canberra February June 2018
Cheng Soon Ong & Christian Walder Research Group and College of Engineering and Computer Science Canberra February June 2018 (Many figures from C. M. Bishop, "Pattern Recognition and ") 1of 305 Part VII
More informationCPSC 540: Machine Learning
CPSC 540: Machine Learning Expectation Maximization Mark Schmidt University of British Columbia Winter 2018 Last Time: Learning with MAR Values We discussed learning with missing at random values in data:
More informationAlgorithmIndependent Learning Issues
AlgorithmIndependent Learning Issues Selim Aksoy Department of Computer Engineering Bilkent University saksoy@cs.bilkent.edu.tr CS 551, Spring 2007 c 2007, Selim Aksoy Introduction We have seen many learning
More informationMachine Learning. RegressionBased Classification & Gaussian Discriminant Analysis. Manfred Huber
Machine Learning RegressionBased Classification & Gaussian Discriminant Analysis Manfred Huber 2015 1 Logistic Regression Linear regression provides a nice representation and an efficient solution to
More informationMachine Learning Lecture 7
Course Outline Machine Learning Lecture 7 Fundamentals (2 weeks) Bayes Decision Theory Probability Density Estimation Statistical Learning Theory 23.05.2016 Discriminative Approaches (5 weeks) Linear Discriminant
More informationGaussian Mixture Models
Gaussian Mixture Models Pradeep Ravikumar Coinstructor: Manuela Veloso Machine Learning 10701 Some slides courtesy of Eric Xing, Carlos Guestrin (One) bad case for K means Clusters may overlap Some
More informationAnnouncements. Proposals graded
Announcements Proposals graded Kevin Jamieson 2018 1 Bayesian Methods Machine Learning CSE546 Kevin Jamieson University of Washington November 1, 2018 2018 Kevin Jamieson 2 MLE Recap  coin flips Data:
More informationCS534: Machine Learning. Thomas G. Dietterich 221C Dearborn Hall
CS534: Machine Learning Thomas G. Dietterich 221C Dearborn Hall tgd@cs.orst.edu http://www.cs.orst.edu/~tgd/classes/534 1 Course Overview Introduction: Basic problems and questions in machine learning.
More informationMachine Learning (CS 567) Lecture 5
Machine Learning (CS 567) Lecture 5 Time: TTh 5:00pm  6:20pm Location: GFS 118 Instructor: Sofus A. Macskassy (macskass@usc.edu) Office: SAL 216 Office hours: by appointment Teaching assistant: Cheol
More informationLatent Variable Models
Latent Variable Models Stefano Ermon, Aditya Grover Stanford University Lecture 5 Stefano Ermon, Aditya Grover (AI Lab) Deep Generative Models Lecture 5 1 / 31 Recap of last lecture 1 Autoregressive models:
More informationExpectation Maximization Algorithm
Expectation Maximization Algorithm Vibhav Gogate The University of Texas at Dallas Slides adapted from Carlos Guestrin, Dan Klein, Luke Zettlemoyer and Dan Weld The Evils of Hard Assignments? Clusters
More informationECE521 week 3: 23/26 January 2017
ECE521 week 3: 23/26 January 2017 Outline Probabilistic interpretation of linear regression  Maximum likelihood estimation (MLE)  Maximum a posteriori (MAP) estimation Biasvariance tradeoff Linear
More informationIntroduction to Machine Learning
Outline Introduction to Machine Learning Bayesian Classification Varun Chandola March 8, 017 1. {circular,large,light,smooth,thick}, malignant. {circular,large,light,irregular,thick}, malignant 3. {oval,large,dark,smooth,thin},
More informationLecture 3: Pattern Classification. Pattern classification
EE E68: Speech & Audio Processing & Recognition Lecture 3: Pattern Classification 3 4 5 The problem of classification Linear and nonlinear classifiers Probabilistic classification Gaussians, mitures and
More informationIntro. ANN & Fuzzy Systems. Lecture 15. Pattern Classification (I): Statistical Formulation
Lecture 15. Pattern Classification (I): Statistical Formulation Outline Statistical Pattern Recognition Maximum Posterior Probability (MAP) Classifier Maximum Likelihood (ML) Classifier KNearest Neighbor
More informationClustering. Professor Ameet Talwalkar. Professor Ameet Talwalkar CS260 Machine Learning Algorithms March 8, / 26
Clustering Professor Ameet Talwalkar Professor Ameet Talwalkar CS26 Machine Learning Algorithms March 8, 217 1 / 26 Outline 1 Administration 2 Review of last lecture 3 Clustering Professor Ameet Talwalkar
More informationECE521 Lecture7. Logistic Regression
ECE521 Lecture7 Logistic Regression Outline Review of decision theory Logistic regression A single neuron Multiclass classification 2 Outline Decision theory is conceptually easy and computationally hard
More informationSupport Vector Machines
Support Vector Machines Le Song Machine Learning I CSE 6740, Fall 2013 Naïve Bayes classifier Still use Bayes decision rule for classification P y x = P x y P y P x But assume p x y = 1 is fully factorized
More informationMIRA, SVM, knn. Lirong Xia
MIRA, SVM, knn Lirong Xia Linear Classifiers (perceptrons) Inputs are feature values Each feature has a weight Sum is the activation activation w If the activation is: Positive: output +1 Negative, output
More informationThe classifier. Theorem. where the min is over all possible classifiers. To calculate the Bayes classifier/bayes risk, we need to know
The Bayes classifier Theorem The classifier satisfies where the min is over all possible classifiers. To calculate the Bayes classifier/bayes risk, we need to know Alternatively, since the maximum it is
More informationThe classifier. Linear discriminant analysis (LDA) Example. Challenges for LDA
The Bayes classifier Linear discriminant analysis (LDA) Theorem The classifier satisfies In linear discriminant analysis (LDA), we make the (strong) assumption that where the min is over all possible classifiers.
More informationUniversität Potsdam Institut für Informatik Lehrstuhl Maschinelles Lernen. Bayesian Learning. Tobias Scheffer, Niels Landwehr
Universität Potsdam Institut für Informatik Lehrstuhl Maschinelles Lernen Bayesian Learning Tobias Scheffer, Niels Landwehr Remember: Normal Distribution Distribution over x. Density function with parameters
More informationParametric Unsupervised Learning Expectation Maximization (EM) Lecture 20.a
Parametric Unsupervised Learning Expectation Maximization (EM) Lecture 20.a Some slides are due to Christopher Bishop Limitations of Kmeans Hard assignments of data points to clusters small shift of a
More informationPattern Recognition and Machine Learning
Christopher M. Bishop Pattern Recognition and Machine Learning ÖSpri inger Contents Preface Mathematical notation Contents vii xi xiii 1 Introduction 1 1.1 Example: Polynomial Curve Fitting 4 1.2 Probability
More informationL11: Pattern recognition principles
L11: Pattern recognition principles Bayesian decision theory Statistical classifiers Dimensionality reduction Clustering This lecture is partly based on [Huang, Acero and Hon, 2001, ch. 4] Introduction
More informationChapter 6 Classification and Prediction (2)
Chapter 6 Classification and Prediction (2) Outline Classification and Prediction Decision Tree Naïve Bayes Classifier Support Vector Machines (SVM) Knearest Neighbors Accuracy and Error Measures Feature
More informationComputer Vision Group Prof. Daniel Cremers. 6. Mixture Models and ExpectationMaximization
Prof. Daniel Cremers 6. Mixture Models and ExpectationMaximization Motivation Often the introduction of latent (unobserved) random variables into a model can help to express complex (marginal) distributions
More informationLinear Classification
Linear Classification Lili MOU moull12@sei.pku.edu.cn http://sei.pku.edu.cn/ moull12 23 April 2015 Outline Introduction Discriminant Functions Probabilistic Generative Models Probabilistic Discriminative
More informationKMeans and Gaussian Mixture Models
KMeans and Gaussian Mixture Models David Rosenberg New York University October 29, 2016 David Rosenberg (New York University) DSGA 1003 October 29, 2016 1 / 42 KMeans Clustering KMeans Clustering David
More informationLinear Classification: Probabilistic Generative Models
Linear Classification: Probabilistic Generative Models Sargur N. University at Buffalo, State University of New York USA 1 Linear Classification using Probabilistic Generative Models Topics 1. Overview
More informationMachine Learning, Fall 2012 Homework 2
060 Machine Learning, Fall 202 Homework 2 Instructors: Tom Mitchell, Ziv BarJoseph TA in charge: Selen Uguroglu email: sugurogl@cs.cmu.edu SOLUTIONS Naive Bayes, 20 points Problem. Basic concepts, 0
More informationCENG 793. On Machine Learning and Optimization. Sinan Kalkan
CENG 793 On Machine Learning and Optimization Sinan Kalkan 2 Now Introduction to ML Problem definition Classes of approaches KNN Support Vector Machines Softmax classification / logistic regression Parzen
More informationEngineering Part IIB: Module 4F10 Statistical Pattern Processing Lecture 5: Single Layer Perceptrons & Estimating Linear Classifiers
Engineering Part IIB: Module 4F0 Statistical Pattern Processing Lecture 5: Single Layer Perceptrons & Estimating Linear Classifiers Phil Woodland: pcw@eng.cam.ac.uk Michaelmas 202 Engineering Part IIB:
More informationStatistical Machine Learning Hilary Term 2018
Statistical Machine Learning Hilary Term 2018 Pier Francesco Palamara Department of Statistics University of Oxford Slide credits and other course material can be found at: http://www.stats.ox.ac.uk/~palamara/sml18.html
More informationLogistic Regression. Seungjin Choi
Logistic Regression Seungjin Choi Department of Computer Science and Engineering Pohang University of Science and Technology 77 Cheongamro, Namgu, Pohang 37673, Korea seungjin@postech.ac.kr http://mlg.postech.ac.kr/
More informationDiscriminative Models
No.5 Discriminative Models Hui Jiang Department of Electrical Engineering and Computer Science Lassonde School of Engineering York University, Toronto, Canada Outline Generative vs. Discriminative models
More informationIntroduction to Machine Learning
1, DATA11002 Introduction to Machine Learning Lecturer: Antti Ukkonen TAs: Saska Dönges and Janne Leppäaho Department of Computer Science University of Helsinki (based in part on material by Patrik Hoyer,
More informationESS2222. Lecture 4 Linear model
ESS2222 Lecture 4 Linear model Hosein Shahnas University of Toronto, Department of Earth Sciences, 1 Outline Logistic Regression Predicting Continuous Target Variables Support Vector Machine (Some Details)
More informationClassification: The rest of the story
U NIVERSITY OF ILLINOIS AT URBANACHAMPAIGN CS598 Machine Learning for Signal Processing Classification: The rest of the story 3 October 2017 Today s lecture Important things we haven t covered yet Fisher
More informationLINEAR MODELS FOR CLASSIFICATION. J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
LINEAR MODELS FOR CLASSIFICATION Classification: Problem Statement 2 In regression, we are modeling the relationship between a continuous input variable x and a continuous target variable t. In classification,
More informationLecture 3. Linear Regression II Bastian Leibe RWTH Aachen
Advanced Machine Learning Lecture 3 Linear Regression II 02.11.2015 Bastian Leibe RWTH Aachen http://www.vision.rwthaachen.de/ leibe@vision.rwthaachen.de This Lecture: Advanced Machine Learning Regression
More informationLast updated: Oct 22, 2012 LINEAR CLASSIFIERS. J. Elder CSE 4404/5327 Introduction to Machine Learning and Pattern Recognition
Last updated: Oct 22, 2012 LINEAR CLASSIFIERS Problems 2 Please do Problem 8.3 in the textbook. We will discuss this in class. Classification: Problem Statement 3 In regression, we are modeling the relationship
More informationStatistical learning. Chapter 20, Sections 1 4 1
Statistical learning Chapter 20, Sections 1 4 Chapter 20, Sections 1 4 1 Outline Bayesian learning Maximum a posteriori and maximum likelihood learning Bayes net learning ML parameter learning with complete
More informationBased on slides by Richard Zemel
CSC 412/2506 Winter 2018 Probabilistic Learning and Reasoning Lecture 3: Directed Graphical Models and Latent Variables Based on slides by Richard Zemel Learning outcomes What aspects of a model can we
More informationLecture 3: Machine learning, classification, and generative models
EE E6820: Speech & Audio Processing & Recognition Lecture 3: Machine learning, classification, and generative models 1 Classification 2 Generative models 3 Gaussian models Michael Mandel
More informationUNIVERSITY of PENNSYLVANIA CIS 520: Machine Learning Final, Fall 2013
UNIVERSITY of PENNSYLVANIA CIS 520: Machine Learning Final, Fall 2013 Exam policy: This exam allows two onepage, twosided cheat sheets; No other materials. Time: 2 hours. Be sure to write your name and
More informationIntroduction to Machine Learning. Maximum Likelihood and Bayesian Inference. Lecturers: Eran Halperin, Yishay Mansour, Lior Wolf
1 Introduction to Machine Learning Maximum Likelihood and Bayesian Inference Lecturers: Eran Halperin, Yishay Mansour, Lior Wolf 201314 We know that X ~ B(n,p), but we do not know p. We get a random sample
More informationParametric Models. Dr. Shuang LIANG. School of Software Engineering TongJi University Fall, 2012
Parametric Models Dr. Shuang LIANG School of Software Engineering TongJi University Fall, 2012 Today s Topics Maximum Likelihood Estimation Bayesian Density Estimation Today s Topics Maximum Likelihood
More informationLecture 12. Neural Networks Bastian Leibe RWTH Aachen
Advanced Machine Learning Lecture 12 Neural Networks 10.12.2015 Bastian Leibe RWTH Aachen http://www.vision.rwthaachen.de/ leibe@vision.rwthaachen.de This Lecture: Advanced Machine Learning Regression
More informationSYDE 372 Introduction to Pattern Recognition. Probability Measures for Classification: Part I
SYDE 372 Introduction to Pattern Recognition Probability Measures for Classification: Part I Alexander Wong Department of Systems Design Engineering University of Waterloo Outline 1 2 3 4 Why use probability
More information5. Discriminant analysis
5. Discriminant analysis We continue from Bayes s rule presented in Section 3 on p. 85 (5.1) where c i is a class, x isapdimensional vector (data case) and we use class conditional probability (density
More informationVariational Autoencoders
Variational Autoencoders Recap: Story so far A classification MLP actually comprises two components A feature extraction network that converts the inputs into linearly separable features Or nearly linearly
More informationσ(a) = a N (x; 0, 1 2 ) dx. σ(a) = Φ(a) =
Until now we have always worked with likelihoods and prior distributions that were conjugate to each other, allowing the computation of the posterior distribution to be done in closed form. Unfortunately,
More informationCS 340 Lec. 18: Multivariate Gaussian Distributions and Linear Discriminant Analysis
CS 3 Lec. 18: Multivariate Gaussian Distributions and Linear Discriminant Analysis AD March 11 AD ( March 11 1 / 17 Multivariate Gaussian Consider data { x i } N i=1 where xi R D and we assume they are
More informationMultivariate statistical methods and data mining in particle physics
Multivariate statistical methods and data mining in particle physics RHUL Physics www.pp.rhul.ac.uk/~cowan Academic Training Lectures CERN 16 19 June, 2008 1 Outline Statement of the problem Some general
More informationComputer Vision Group Prof. Daniel Cremers. 2. Regression (cont.)
Prof. Daniel Cremers 2. Regression (cont.) Regression with MLE (Rep.) Assume that y is affected by Gaussian noise : t = f(x, w)+ where Thus, we have p(t x, w, )=N (t; f(x, w), 2 ) 2 Maximum APosteriori
More informationIntroduction to Machine Learning
Introduction to Machine Learning Thomas G. Dietterich tgd@eecs.oregonstate.edu 1 Outline What is Machine Learning? Introduction to Supervised Learning: Linear Methods Overfitting, Regularization, and the
More informationLecture 2. G. Cowan Lectures on Statistical Data Analysis Lecture 2 page 1
Lecture 2 1 Probability (90 min.) Definition, Bayes theorem, probability densities and their properties, catalogue of pdfs, Monte Carlo 2 Statistical tests (90 min.) general concepts, test statistics,
More informationLecture 4: Probabilistic Learning
DD2431 Autumn, 2015 1 Maximum Likelihood Methods Maximum A Posteriori Methods Bayesian methods 2 Classification vs Clustering Heuristic Example: Kmeans Expectation Maximization 3 Maximum Likelihood Methods
More informationDiscriminative Models
No.5 Discriminative Models Hui Jiang Department of Electrical Engineering and Computer Science Lassonde School of Engineering York University, Toronto, Canada Outline Generative vs. Discriminative models
More informationCurve Fitting Revisited, Bishop1.2.5
Curve Fitting Revisited, Bishop1.2.5 Maximum Likelihood Bishop 1.2.5 Model Likelihood differentiation p(t x, w, β) = Maximum Likelihood N N ( t n y(x n, w), β 1). (1.61) n=1 As we did in the case of the
More information