Logistic Regression and Generalized Linear Models
|
|
- Buck Webster
- 6 years ago
- Views:
Transcription
1 Logistic Regression and Generalized Linear Models Sridhar Mahadevan University of Massachusetts Sridhar Mahadevan: CMPSCI 689 p. 1/2
2 Topics Generative vs. Discriminative models In many cases, it is difficult to model data using a parametric class conditional density P(X ω, θ) Yet, in many problems, a linear decision boundary is usually adequate to separate classes (also, gaussian densities with a shared covariance matrix produces a linear decision boundary). Logistic regression: discriminative model for classification that produces linear decision boundaries Model fitting problem solved using maximum likelihood Iterative gradient-based algorithm for solving nonlinear maximum likelihood equations Recursive weighted least squares regression Logistic regression is an instance of a generalized linear model (GLM), which consists of a large variety of exponential models.glms can also be extended to generalized additive models (GAMs). Sridhar Mahadevan: CMPSCI 689 p. 2/2
3 Discriminative vs. Generative Models Both generative and discriminative approaches address the problem of modeling the discriminant function P(y x) of output labels (or values) y conditioned on the input x. In generative models, we estimate both P(x) and P(x y), and use Bayes rule to compute the discriminant. P(y x) P(x)P(x y) Discriminative approaches model the conditional distribution P(y x) directly, and ignore the marginal P(x). We now turn to explore several types instances of discriminative models, including logistic regression in this class, and later several other types including support vector machines. Sridhar Mahadevan: CMPSCI 689 p. 3/2
4 Generalized Linear Models In linear regression, we model the output y as a linear function of the input variables, with a noise term that is zero mean constant variance Gaussian. y = g(x) + ǫ, where the conditional mean E(y x) = g(x), and the noise term is ǫ. g(x) = β T x (where β 0 is an offset term). We saw earlier that the maximum likelihood framework justified the use of a squared error loss function, provided the errors were IID gaussian (the variance does not matter). We want to generalize this idea of specifying a model family by specifying the type of error distribution: When the output variable y is discrete (e.g., binary or multinomial), the noise term is not gaussian, but binomial or multinomial. A change in the mean is coupled by a change in the variance, and we want to be able to couple mean and variance in our model. Generalized linear models provides a rich tool of models based on specifying the error distribution. Sridhar Mahadevan: CMPSCI 689 p. 4/2
5 Logit Function Since the output variable y only takes on values (0, 1) (for binary classification), we need a different way of representing E(y x) so that the range of y (0, 1). One convenient form to use is the sigmoid or logistic function. Let us assume a vector-valued input variable x = (x 1,..., x p ). The logistic function is S shaped and approaches 0 (as x ) or 1 (as x ). P(y = 1 x, β) = µ(x β) = eβt x 1 + e βt x = e βt x P(y = 0 x, β) = 1 µ(x β) = e βt x We assume an extra input x 0 = 1, so that β 0 is an offset. We can invert the above transformation to get the logit function g(x β) = log µ(x β) 1 µ(x β) = βt x Sridhar Mahadevan: CMPSCI 689 p. 5/2
6 Logistic Regression y β2 X2 X1 β1 β0 X0 Sridhar Mahadevan: CMPSCI 689 p. 6/2
7 Example Dataset for Logistic Regression The data set we are analyzing is coronary heart disease in South Africa. The chd response (output) variable is binary (yes, no), and there are 9 predictor variables: There are 462 instances, out of which 160 are cases (positive instances), and 302 are controls (negative instances). The predictor variables are systolic blood pressure, tobacco, ldl, famhist, obesity, alcohol, age, adiposity, typea, Let s focus on a subset of the predictors: sbp, tobacco, ldl, famhist, obesity, alcohol, age. We want to fit a model of the following form P(chd = 1 x, β) = e βt x where β T x = β 0 +β 1 x sbp +β 2 x tobacco +β 3 x ldl +β 4 x famhist +β 5 x age +β 6 x alcohol +β 7 x obesity Sridhar Mahadevan: CMPSCI 689 p. 7/2
8 Noise Model for Logistic Regression Let us try to represent the logistic regression model as y = µ(x β) + ǫ and ask ourself what sort of noise model is represented by ǫ. Since y takes on the value 1 with probability µ(x β), it follows that ǫ can also only take on two possible values, namely If y = 1, then ǫ = 1 µ(x β) with probability µ(x β). Conversely, if y = 0, then ǫ = µ(x β) and this happens with probability (1 µ(x β)). This analysis shows that the error term in logistic regression is a binomially distributed random variable. Its moments can be computed readily as shown below: E(ǫ) = µ(x β)(1 µ(x β)) (1 µ(x β))µ(x β) = 0 (the error term has mean 0). V ar(ǫ) = Eǫ 2 (Eǫ) 2 = Eǫ 2 = µ(x β)(1 µ(x β)) (show this!) Sridhar Mahadevan: CMPSCI 689 p. 8/2
9 Maximum Likelihood for LR Suppose we want to fit a logistic regression model to a dataset of n observations X = (x 1, y 1 ),..., (x n, y n ). We can express the conditional likelihood of a single observation simply as P(y i x i, β) = µ(x i β) yi (1 µ(x i β)) 1 yi Hence, the conditional likelihood of the entire dataset can be written as P(Y X, β) = n i=1 µ(x i β) yi (1 µ(x i β)) 1 yi The conditional log-likelihood is then simply l(β X, Y ) = n i=1 y i log µ(x i β) + (1 y i )log(1 µ(x i β)) Sridhar Mahadevan: CMPSCI 689 p. 9/2
10 Maximum Likelihood for LR We solve the conditional log-likelihood equation by taking gradients l(β X, Y ) β k = n i=1 y i 1 µ(x i β) µ(x i β) β k (1 y i ) 1 (1 µ(x i β)) µ(x i ) β k Using the fact that µ(xi β) β k = 1 ( β k 1+e βt x i ) = µ(x i β)(1 µ(x i β))x i k, we get l(β X, Y ) β k = n i=1 x i k (yi µ(x i β)) Setting this to 0, since x 0 = 1 the first component of these equations reduces to n i=1 y i = n i=1 µ(x i β) The expected number of instances of each class must match the observed number. Sridhar Mahadevan: CMPSCI 689 p. 10/2
11 Newton-Raphson Method Newton s method is a general procedure for finding the roots of an equation f(θ) = 0. Newton s algorithm is based on the recursion θ t+1 = θ t f(θ t) f (θ t ) Newton s method finds the minimum of a function f. We want to find the maximum of the log likehood equation. But, the maximum of a function f(θ) is exactly when its derivative f (θ) = 0. So, plugging in f (θ) for f(θ) above, we get θ t+1 = θ t f (θ t ) f (θ t ) Sridhar Mahadevan: CMPSCI 689 p. 11/2
12 Fisher Scoring In logistic regresion, the parameter β is a vector, so we have to use the Newton-Raphson algorithm β t+1 = β t H 1 β l(β t X, Y ) Here, β l(β t X, Y ) is the vector of partial derivatives of the log-likelihood equation H ij = 2 l(β X,Y ) β i β j is the Hessian matrix of second order derivatives. The use of Newton s method to find the solution to the conditional log-likelihood equation is called Fisher scoring. Sridhar Mahadevan: CMPSCI 689 p. 12/2
13 Fisher Scoring for Maximum Likelihood Taking the second derivative of the likelihood score equations gives us 2 l(β X, Y ) β k β m = n i=1 x i k xi mµ(x i β)(1 µ(x i β)) We can use matrix notation to write the Newton-Raphson algorithm for logistic regression. Define the n n diagonal matrix W = µ(x 1 β)(1 µ(x 1 β)) µ(x 2 β)(1 µ(x 2 β)) µ(x n β)(1 µ(x n β)) Let Y be an n 1 column vector of output values, and X be the design matrix of size n (p + 1) of input values, and P be the column vector of fitted probability values µ(x i β). Sridhar Mahadevan: CMPSCI 689 p. 13/2
14 Iterative Weighted Least Squares The gradient of the log likelihood can be written in matrix form as l(β X, Y ) β = n i=1 x i (y i µ(x i β)) = X T (Y P) The Hessian can be written as 2 l(β X, Y ) β β T = X T WX The Newton-Raphson algorithm then becomes β new = β old + (X T WX) 1 X T (Y P) = (X T WX) 1 X T W ( Xβ old + W 1 (Y P) ) = (X T WX) 1 X T WZ where Z Xβ old + W 1 (Y P) Sridhar Mahadevan: CMPSCI 689 p. 14/2
15 Weighted Least Squares Regression Weighted least squares regression finds the best least-squares solution to the equation WAx Wb (WA) T WAˆx = (WA) T Wb ˆx = (A T CA) 1 A T Cb where C = W T W Returning to logistic regression, we now see β new = (X T WX) 1 X T WZ is weighted least squares regression (where X is the matrix A above, W is a diagonal weight vector with entries µ(x i β)(1 µ(x i β)), and Z corresponds to the vector b above). It is termed recursive weighted least squares, because at each step, the weight vector W keeps changing (since the β s are changing). We can visualize RWLS as solving the following equation β new argmin β (Z Xβ) T W(Z Xβ) Sridhar Mahadevan: CMPSCI 689 p. 15/2
16 Stochastic Gradient Ascent Newton s method is often referred to as a 2 nd order method, because it involves taking the Hessian. This can be difficult in large problems, because it involves matrix inversion. One way to avoid this is to settle for slower convergence, but less work at each step. For each training instance (x, y) we can derive an incremental gradient update rule. l(β x, y) β j = x j (y µ(x β)) The stochastic gradient ascent rule can be written as (for instance (x i, y i )) β j β j + α(y i µ(x i β))x i j The convergence of this update rule can be iffy, unlike Newton s method. It depends on tweaking the learning rate α so that the steps are not too small or too large, and also a cooling schedule. Sridhar Mahadevan: CMPSCI 689 p. 16/2
17 The LMS Algorithm The LMS (least mean square) algorithm can be used to solve least squares regression problems incrementally taking the gradient of the loss function w.r.t. the parameters, and adjusting the weights for each data instance. h(x β) = β 0 + n j=1 β j x j = β T x Given a data set D, we want to find the vector β that minimizes the (mean) squared error loss L(h) = 1 2 i (y i h(x; β)) 2. LMS algorithm: find the gradient on a particular instance L(h) β j = (y h(x β))x j Adjust the weight in the direction of decreasing the error (negative gradient). β j β j + α(y h(x β))x j Sridhar Mahadevan: CMPSCI 689 p. 17/2
18 Logistic Regression vs LDA Recall from Bayes decision theory that when the class conditional densities P(x ω i, µ i,σ) share the same underlying covariance matrix Σ, the decision boundary that separates the classes is a line (or hyperplane). Such Bayesian classifiers are called linear discriminant classifiers. We see from above that logistic regression also produces a decision boundary that is a hyperplane, since its discriminant function was shown to be g(x β) = log µ(x β) 1 µ(x β) = βt x So, if both gaussian LDA and logistic regression produce linear decision boundaries, which is preferable? As a general rule, logistic regression works in a larger class of problems because it does not assume the underlying class conditional densities are Gaussian! Sridhar Mahadevan: CMPSCI 689 p. 18/2
19 Generalized Linear Models A generalized linear model is specified using two functions A link function that describes how the mean depends on the linear predictor g(µ) = η, where η = β T x. For example, in logistic regression, the link function is the logit function g(µ) = log µ 1 µ = βt x In linear regression, the link function is simply g(µ) = µ The inverse of the logit function g 1 (η) = µ describes how the mean µ can be related back to the linear predictor. For logistic regression, the inverse link function is the sigmoid. A variance function that specifies how the variance of the output y depends on the mean φv (µ), where φ is constant. For example, in logistic regression, the variance is µ(1 µ), because it is a binomial distribution. In linear regression, the variance is simply 1 because the error term is modeled as having constant variance. Sridhar Mahadevan: CMPSCI 689 p. 19/2
20 Generalized Linear Models Distribution Link Function Variance Function Gaussian µ 1 Binomial log µ 1 µ µ(1 µ) Poisson log µ µ Gamma 1 µ µ 2 Sridhar Mahadevan: CMPSCI 689 p. 20/2
21 Multiway Classification As one more example of a generalized linear model, we generalize the logistic regression model to a multinomial (e.g, sorting into various categories) It easily follows that log P(Y = 1 X = x, β) log P(Y = k X = x, β) P(Y = 2 X = x, β) log P(Y = k X = x, β) P(Y = K 1 X = x, β) P(Y = k X = x, β) = β1 T x = β2 T x... = βk 1 T x P(Y = k X = x, β) = P(Y = K X = x, β) = e βt k x 1 + K 1 l=1 eβt l x, k = 1,..., K K 1 l=1 eβt l x, k = 1,..., K 1 Sridhar Mahadevan: CMPSCI 689 p. 21/2
22 Multinomial Link function The multinomial PDF can be written as a member of the exponential family P(y φ) = φ 1{y=1} 1 φ 2{y=2} 2... φ 1{y=K} K = φ y 1 1 φ = e ηt ỹ a(η) k 1 y φ1 K i=1 ỹi The last expression above is an instance of the generic form of the exponential family (check Weber handout or earlier notes) Instead of a single link function, we now have a vector η = log φ 1 φ k log φ 2 φ k. log φ k 1 φ k Sridhar Mahadevan: CMPSCI 689 p. 22/2
23 Fitting GLM Models in R The statisics package R has comprehensive built-in features for fitting generalized linear models (and many related models as well). This is a very brief intro to model fitting in R. See the R documentation, as well as the excellent text Statistical Models in S by Chambers and Hastie. The main function is called glm(formula=, family=,...). Here, formula is a symbolic description of the model that is to be fit. family is the description of the appropriate error distribution and link function to be used. Sridhar Mahadevan: CMPSCI 689 p. 23/2
24 Heart Disease Dataset The data set we are analyzing is coronary heart disease in South Africa. The chd response (output) variable is binary (yes, no), and there are 9 predictor variables: systolic blood pressure, tobacco, ldl, famhist, obesity, alcohol, age, adiposity, typea, The R command glm(chd sbp + tobacco + ldl + famhist + age + alcohol + obesity, family = binomial, data = heart) will fit a logistic regression model to the heart disease data set (assuming the dataset is loaded into the system). The R command glm(chd sbp + tobacco + ldl + famhist + age + alcohol + obesity, family = binomial(link=probit), data = heart) will fit the same data, now using the probit distribution (inverse CDF of the normal distribution). Sridhar Mahadevan: CMPSCI 689 p. 24/2
25 Heart Disease Data The following model was found to be the best fit to the data, using maximum likelihood. Coefficient Estimate (Intercept) sbp tobacco ldl famhistpresent age alcohol obesity Sridhar Mahadevan: CMPSCI 689 p. 25/2
26 Polynomial Regression Load the data data(cars93) which contains information about 93 models of cars. names(cars93) tells you the attributes that make up the dataset. [1] "Manufacturer" "Model" "Type" [4] "Min.Price" "Price" "Max.Price" [7] "MPG.city" "MPG.highway" "AirBags" [10] "DriveTrain" "Cylinders" "EngineSize" [13] "Horsepower" "RPM" "Rev.per.mile" [16] "Man.trans.avail" "Fuel.tank.capacity" "Passengers" [19] "Length" "Wheelbase" "Width" [22] "Turn.circle" "Rear.seat.room" "Luggage.room" [25] "Weight" "Origin" "Make" Let s assume we want to fit a polynomial regression model, and we chose Price as the predictor variable, and Weight as the response variable. We can graph the these two variables using: plot(weight Price, data = Cars93) Sridhar Mahadevan: CMPSCI 689 p. 26/2
27 Fitting GLM Models in R: Cars W P Sridhar Mahadevan: CMPSCI 689 p. 27/2
28 Fitting GLM Models in R The R command to fit a d th degree polynomial is glm(weight poly(price, d), family = gaussian, data = Cars93) Notice how we specified the link function to be gaussian. W P Sridhar Mahadevan: CMPSCI 689 p. 28/2
29 Generalized Additive Models In the regression setting, a generalized additive model has the form E(Y X 1,..., X p ) = α + f 1 (X 1 ) + f 2 (X 2 ) f p (X p ) Here, the f i are unspecified smooth (nonparametric ) functions. In the classification setting, a generalized additive logistic regression model has the form log( µ(x β) 1 µ(x β) = α + f 1(X 1 ) + f 2 (X 2 ) f p (X p ) See Section in Hastie s book (Statistical Learning) for how to fit additive logistic regression models to an -spam dataset. Sridhar Mahadevan: CMPSCI 689 p. 29/2
MS&E 226: Small Data
MS&E 226: Small Data Lecture 12: Logistic regression (v1) Ramesh Johari ramesh.johari@stanford.edu Fall 2015 1 / 30 Regression methods for binary outcomes 2 / 30 Binary outcomes For the duration of this
More informationMachine Learning. Lecture 3: Logistic Regression. Feng Li.
Machine Learning Lecture 3: Logistic Regression Feng Li fli@sdu.edu.cn https://funglee.github.io School of Computer Science and Technology Shandong University Fall 2016 Logistic Regression Classification
More informationAdministration. Homework 1 on web page, due Feb 11 NSERC summer undergraduate award applications due Feb 5 Some helpful books
STA 44/04 Jan 6, 00 / 5 Administration Homework on web page, due Feb NSERC summer undergraduate award applications due Feb 5 Some helpful books STA 44/04 Jan 6, 00... administration / 5 STA 44/04 Jan 6,
More informationMS&E 226: Small Data
MS&E 226: Small Data Lecture 9: Logistic regression (v2) Ramesh Johari ramesh.johari@stanford.edu 1 / 28 Regression methods for binary outcomes 2 / 28 Binary outcomes For the duration of this lecture suppose
More informationClassification. Chapter Introduction. 6.2 The Bayes classifier
Chapter 6 Classification 6.1 Introduction Often encountered in applications is the situation where the response variable Y takes values in a finite set of labels. For example, the response Y could encode
More informationSTA 450/4000 S: January
STA 450/4000 S: January 6 005 Notes Friday tutorial on R programming reminder office hours on - F; -4 R The book Modern Applied Statistics with S by Venables and Ripley is very useful. Make sure you have
More informationOutline. Supervised Learning. Hong Chang. Institute of Computing Technology, Chinese Academy of Sciences. Machine Learning Methods (Fall 2012)
Outline Hong Chang Institute of Computing Technology, Chinese Academy of Sciences Machine Learning Methods (Fall 2012) Outline Outline I 1 Linear Models for Regression Linear Regression Probabilistic Interpretation
More informationECLT 5810 Linear Regression and Logistic Regression for Classification. Prof. Wai Lam
ECLT 5810 Linear Regression and Logistic Regression for Classification Prof. Wai Lam Linear Regression Models Least Squares Input vectors is an attribute / feature / predictor (independent variable) The
More informationLecture 5: LDA and Logistic Regression
Lecture 5: and Logistic Regression Hao Helen Zhang Hao Helen Zhang Lecture 5: and Logistic Regression 1 / 39 Outline Linear Classification Methods Two Popular Linear Models for Classification Linear Discriminant
More informationMachine Learning Linear Classification. Prof. Matteo Matteucci
Machine Learning Linear Classification Prof. Matteo Matteucci Recall from the first lecture 2 X R p Regression Y R Continuous Output X R p Y {Ω 0, Ω 1,, Ω K } Classification Discrete Output X R p Y (X)
More informationECLT 5810 Linear Regression and Logistic Regression for Classification. Prof. Wai Lam
ECLT 5810 Linear Regression and Logistic Regression for Classification Prof. Wai Lam Linear Regression Models Least Squares Input vectors is an attribute / feature / predictor (independent variable) The
More informationMATH 829: Introduction to Data Mining and Analysis Logistic regression
1/11 MATH 829: Introduction to Data Mining and Analysis Logistic regression Dominique Guillot Departments of Mathematical Sciences University of Delaware March 7, 2016 Logistic regression 2/11 Suppose
More informationLinear Methods for Prediction
Chapter 5 Linear Methods for Prediction 5.1 Introduction We now revisit the classification problem and focus on linear methods. Since our prediction Ĝ(x) will always take values in the discrete set G we
More informationMachine Learning 2017
Machine Learning 2017 Volker Roth Department of Mathematics & Computer Science University of Basel 21st March 2017 Volker Roth (University of Basel) Machine Learning 2017 21st March 2017 1 / 41 Section
More informationLinear model A linear model assumes Y X N(µ(X),σ 2 I), And IE(Y X) = µ(x) = X β, 2/52
Statistics for Applications Chapter 10: Generalized Linear Models (GLMs) 1/52 Linear model A linear model assumes Y X N(µ(X),σ 2 I), And IE(Y X) = µ(x) = X β, 2/52 Components of a linear model The two
More informationLinear Regression With Special Variables
Linear Regression With Special Variables Junhui Qian December 21, 2014 Outline Standardized Scores Quadratic Terms Interaction Terms Binary Explanatory Variables Binary Choice Models Standardized Scores:
More informationContents Lecture 4. Lecture 4 Linear Discriminant Analysis. Summary of Lecture 3 (II/II) Summary of Lecture 3 (I/II)
Contents Lecture Lecture Linear Discriminant Analysis Fredrik Lindsten Division of Systems and Control Department of Information Technology Uppsala University Email: fredriklindsten@ituuse Summary of lecture
More informationApplied Statistics. J. Blanchet and J. Wadsworth. Institute of Mathematics, Analysis, and Applications EPF Lausanne
Applied Statistics J. Blanchet and J. Wadsworth Institute of Mathematics, Analysis, and Applications EPF Lausanne An MSc Course for Applied Mathematicians, Fall 2012 Outline 1 Motivation: Why Applied Statistics?
More informationLecture 5: Linear models for classification. Logistic regression. Gradient Descent. Second-order methods.
Lecture 5: Linear models for classification. Logistic regression. Gradient Descent. Second-order methods. Linear models for classification Logistic regression Gradient descent and second-order methods
More informationLinear Regression Models P8111
Linear Regression Models P8111 Lecture 25 Jeff Goldsmith April 26, 2016 1 of 37 Today s Lecture Logistic regression / GLMs Model framework Interpretation Estimation 2 of 37 Linear regression Course started
More informationLecture 4: Types of errors. Bayesian regression models. Logistic regression
Lecture 4: Types of errors. Bayesian regression models. Logistic regression A Bayesian interpretation of regularization Bayesian vs maximum likelihood fitting more generally COMP-652 and ECSE-68, Lecture
More informationLinear Models in Machine Learning
CS540 Intro to AI Linear Models in Machine Learning Lecturer: Xiaojin Zhu jerryzhu@cs.wisc.edu We briefly go over two linear models frequently used in machine learning: linear regression for, well, regression,
More informationStatistical Data Mining and Machine Learning Hilary Term 2016
Statistical Data Mining and Machine Learning Hilary Term 2016 Dino Sejdinovic Department of Statistics Oxford Slides and other materials available at: http://www.stats.ox.ac.uk/~sejdinov/sdmml Naïve Bayes
More informationGenerative v. Discriminative classifiers Intuition
Logistic Regression Machine Learning 070/578 Carlos Guestrin Carnegie Mellon University September 24 th, 2007 Generative v. Discriminative classifiers Intuition Want to Learn: h:x a Y X features Y target
More informationSTAT5044: Regression and Anova
STAT5044: Regression and Anova Inyoung Kim 1 / 15 Outline 1 Fitting GLMs 2 / 15 Fitting GLMS We study how to find the maxlimum likelihood estimator ˆβ of GLM parameters The likelihood equaions are usually
More informationProbabilistic classification CE-717: Machine Learning Sharif University of Technology. M. Soleymani Fall 2016
Probabilistic classification CE-717: Machine Learning Sharif University of Technology M. Soleymani Fall 2016 Topics Probabilistic approach Bayes decision theory Generative models Gaussian Bayes classifier
More informationLINEAR MODELS FOR CLASSIFICATION. J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
LINEAR MODELS FOR CLASSIFICATION Classification: Problem Statement 2 In regression, we are modeling the relationship between a continuous input variable x and a continuous target variable t. In classification,
More informationOverfitting, Bias / Variance Analysis
Overfitting, Bias / Variance Analysis Professor Ameet Talwalkar Professor Ameet Talwalkar CS260 Machine Learning Algorithms February 8, 207 / 40 Outline Administration 2 Review of last lecture 3 Basic
More informationLinear Classification. CSE 6363 Machine Learning Vassilis Athitsos Computer Science and Engineering Department University of Texas at Arlington
Linear Classification CSE 6363 Machine Learning Vassilis Athitsos Computer Science and Engineering Department University of Texas at Arlington 1 Example of Linear Classification Red points: patterns belonging
More informationCOMS 4771 Introduction to Machine Learning. James McInerney Adapted from slides by Nakul Verma
COMS 4771 Introduction to Machine Learning James McInerney Adapted from slides by Nakul Verma Announcements HW1: Please submit as a group Watch out for zero variance features (Q5) HW2 will be released
More informationLogistic Regression. Seungjin Choi
Logistic Regression Seungjin Choi Department of Computer Science and Engineering Pohang University of Science and Technology 77 Cheongam-ro, Nam-gu, Pohang 37673, Korea seungjin@postech.ac.kr http://mlg.postech.ac.kr/
More informationLecture 4 Discriminant Analysis, k-nearest Neighbors
Lecture 4 Discriminant Analysis, k-nearest Neighbors Fredrik Lindsten Division of Systems and Control Department of Information Technology Uppsala University. Email: fredrik.lindsten@it.uu.se fredrik.lindsten@it.uu.se
More informationMachine Learning, Fall 2012 Homework 2
0-60 Machine Learning, Fall 202 Homework 2 Instructors: Tom Mitchell, Ziv Bar-Joseph TA in charge: Selen Uguroglu email: sugurogl@cs.cmu.edu SOLUTIONS Naive Bayes, 20 points Problem. Basic concepts, 0
More information> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE GRAVIS 2016 BASEL. Logistic Regression. Pattern Recognition 2016 Sandro Schönborn University of Basel
Logistic Regression Pattern Recognition 2016 Sandro Schönborn University of Basel Two Worlds: Probabilistic & Algorithmic We have seen two conceptual approaches to classification: data class density estimation
More informationLast updated: Oct 22, 2012 LINEAR CLASSIFIERS. J. Elder CSE 4404/5327 Introduction to Machine Learning and Pattern Recognition
Last updated: Oct 22, 2012 LINEAR CLASSIFIERS Problems 2 Please do Problem 8.3 in the textbook. We will discuss this in class. Classification: Problem Statement 3 In regression, we are modeling the relationship
More informationCh 4. Linear Models for Classification
Ch 4. Linear Models for Classification Pattern Recognition and Machine Learning, C. M. Bishop, 2006. Department of Computer Science and Engineering Pohang University of Science and echnology 77 Cheongam-ro,
More informationStatistical Machine Learning Hilary Term 2018
Statistical Machine Learning Hilary Term 2018 Pier Francesco Palamara Department of Statistics University of Oxford Slide credits and other course material can be found at: http://www.stats.ox.ac.uk/~palamara/sml18.html
More informationMath for Machine Learning Open Doors to Data Science and Artificial Intelligence. Richard Han
Math for Machine Learning Open Doors to Data Science and Artificial Intelligence Richard Han Copyright 05 Richard Han All rights reserved. CONTENTS PREFACE... - INTRODUCTION... LINEAR REGRESSION... 4 LINEAR
More informationGenerative v. Discriminative classifiers Intuition
Logistic Regression Machine Learning 10701/15781 Carlos Guestrin Carnegie Mellon University September 24 th, 2007 1 Generative v. Discriminative classifiers Intuition Want to Learn: h:x a Y X features
More informationData Mining Stat 588
Data Mining Stat 588 Lecture 9: Basis Expansions Department of Statistics & Biostatistics Rutgers University Nov 01, 2011 Regression and Classification Linear Regression. E(Y X) = f(x) We want to learn
More informationClassification CE-717: Machine Learning Sharif University of Technology. M. Soleymani Fall 2012
Classification CE-717: Machine Learning Sharif University of Technology M. Soleymani Fall 2012 Topics Discriminant functions Logistic regression Perceptron Generative models Generative vs. discriminative
More informationGaussian and Linear Discriminant Analysis; Multiclass Classification
Gaussian and Linear Discriminant Analysis; Multiclass Classification Professor Ameet Talwalkar Slide Credit: Professor Fei Sha Professor Ameet Talwalkar CS260 Machine Learning Algorithms October 13, 2015
More informationSTA216: Generalized Linear Models. Lecture 1. Review and Introduction
STA216: Generalized Linear Models Lecture 1. Review and Introduction Let y 1,..., y n denote n independent observations on a response Treat y i as a realization of a random variable Y i In the general
More informationChap 2. Linear Classifiers (FTH, ) Yongdai Kim Seoul National University
Chap 2. Linear Classifiers (FTH, 4.1-4.4) Yongdai Kim Seoul National University Linear methods for classification 1. Linear classifiers For simplicity, we only consider two-class classification problems
More informationCOMS 4771 Regression. Nakul Verma
COMS 4771 Regression Nakul Verma Last time Support Vector Machines Maximum Margin formulation Constrained Optimization Lagrange Duality Theory Convex Optimization SVM dual and Interpretation How get the
More informationSB1a Applied Statistics Lectures 9-10
SB1a Applied Statistics Lectures 9-10 Dr Geoff Nicholls Week 5 MT15 - Natural or canonical) exponential families - Generalised Linear Models for data - Fitting GLM s to data MLE s Iteratively Re-weighted
More informationComments. x > w = w > x. Clarification: this course is about getting you to be able to think as a machine learning expert
Logistic regression Comments Mini-review and feedback These are equivalent: x > w = w > x Clarification: this course is about getting you to be able to think as a machine learning expert There has to be
More informationSTA 4273H: Statistical Machine Learning
STA 4273H: Statistical Machine Learning Russ Salakhutdinov Department of Statistics! rsalakhu@utstat.toronto.edu! http://www.utstat.utoronto.ca/~rsalakhu/ Sidney Smith Hall, Room 6002 Lecture 3 Linear
More informationLast Time. Today. Bayesian Learning. The Distributions We Love. CSE 446 Gaussian Naïve Bayes & Logistic Regression
CSE 446 Gaussian Naïve Bayes & Logistic Regression Winter 22 Dan Weld Learning Gaussians Naïve Bayes Last Time Gaussians Naïve Bayes Logistic Regression Today Some slides from Carlos Guestrin, Luke Zettlemoyer
More informationMachine Learning: Assignment 1
10-701 Machine Learning: Assignment 1 Due on Februrary 0, 014 at 1 noon Barnabas Poczos, Aarti Singh Instructions: Failure to follow these directions may result in loss of points. Your solutions for this
More informationParametric Models. Dr. Shuang LIANG. School of Software Engineering TongJi University Fall, 2012
Parametric Models Dr. Shuang LIANG School of Software Engineering TongJi University Fall, 2012 Today s Topics Maximum Likelihood Estimation Bayesian Density Estimation Today s Topics Maximum Likelihood
More informationEXAM IN STATISTICAL MACHINE LEARNING STATISTISK MASKININLÄRNING
EXAM IN STATISTICAL MACHINE LEARNING STATISTISK MASKININLÄRNING DATE AND TIME: August 30, 2018, 14.00 19.00 RESPONSIBLE TEACHER: Niklas Wahlström NUMBER OF PROBLEMS: 5 AIDING MATERIAL: Calculator, mathematical
More informationLinear Decision Boundaries
Linear Decision Boundaries A basic approach to classification is to find a decision boundary in the space of the predictor variables. The decision boundary is often a curve formed by a regression model:
More informationMidterm. Introduction to Machine Learning. CS 189 Spring Please do not open the exam before you are instructed to do so.
CS 89 Spring 07 Introduction to Machine Learning Midterm Please do not open the exam before you are instructed to do so. The exam is closed book, closed notes except your one-page cheat sheet. Electronic
More informationLinear & nonlinear classifiers
Linear & nonlinear classifiers Machine Learning Hamid Beigy Sharif University of Technology Fall 1394 Hamid Beigy (Sharif University of Technology) Linear & nonlinear classifiers Fall 1394 1 / 34 Table
More informationLOGISTIC REGRESSION Joseph M. Hilbe
LOGISTIC REGRESSION Joseph M. Hilbe Arizona State University Logistic regression is the most common method used to model binary response data. When the response is binary, it typically takes the form of
More informationPh.D. Qualifying Exam Friday Saturday, January 3 4, 2014
Ph.D. Qualifying Exam Friday Saturday, January 3 4, 2014 Put your solution to each problem on a separate sheet of paper. Problem 1. (5166) Assume that two random samples {x i } and {y i } are independently
More informationMachine Learning. Regression-Based Classification & Gaussian Discriminant Analysis. Manfred Huber
Machine Learning Regression-Based Classification & Gaussian Discriminant Analysis Manfred Huber 2015 1 Logistic Regression Linear regression provides a nice representation and an efficient solution to
More informationSome explanations about the IWLS algorithm to fit generalized linear models
Some explanations about the IWLS algorithm to fit generalized linear models Christophe Dutang To cite this version: Christophe Dutang. Some explanations about the IWLS algorithm to fit generalized linear
More informationMachine learning comes from Bayesian decision theory in statistics. There we want to minimize the expected value of the loss function.
Bayesian learning: Machine learning comes from Bayesian decision theory in statistics. There we want to minimize the expected value of the loss function. Let y be the true label and y be the predicted
More informationThe classifier. Theorem. where the min is over all possible classifiers. To calculate the Bayes classifier/bayes risk, we need to know
The Bayes classifier Theorem The classifier satisfies where the min is over all possible classifiers. To calculate the Bayes classifier/bayes risk, we need to know Alternatively, since the maximum it is
More informationThe classifier. Linear discriminant analysis (LDA) Example. Challenges for LDA
The Bayes classifier Linear discriminant analysis (LDA) Theorem The classifier satisfies In linear discriminant analysis (LDA), we make the (strong) assumption that where the min is over all possible classifiers.
More informationMachine Learning Basics Lecture 2: Linear Classification. Princeton University COS 495 Instructor: Yingyu Liang
Machine Learning Basics Lecture 2: Linear Classification Princeton University COS 495 Instructor: Yingyu Liang Review: machine learning basics Math formulation Given training data x i, y i : 1 i n i.i.d.
More informationKernel Logistic Regression and the Import Vector Machine
Kernel Logistic Regression and the Import Vector Machine Ji Zhu and Trevor Hastie Journal of Computational and Graphical Statistics, 2005 Presented by Mingtao Ding Duke University December 8, 2011 Mingtao
More informationSGN (4 cr) Chapter 5
SGN-41006 (4 cr) Chapter 5 Linear Discriminant Analysis Jussi Tohka & Jari Niemi Department of Signal Processing Tampere University of Technology January 21, 2014 J. Tohka & J. Niemi (TUT-SGN) SGN-41006
More informationLecture 4: Newton s method and gradient descent
Lecture 4: Newton s method and gradient descent Newton s method Functional iteration Fitting linear regression Fitting logistic regression Prof. Yao Xie, ISyE 6416, Computational Statistics, Georgia Tech
More information10701/15781 Machine Learning, Spring 2007: Homework 2
070/578 Machine Learning, Spring 2007: Homework 2 Due: Wednesday, February 2, beginning of the class Instructions There are 4 questions on this assignment The second question involves coding Do not attach
More informationSTA414/2104. Lecture 11: Gaussian Processes. Department of Statistics
STA414/2104 Lecture 11: Gaussian Processes Department of Statistics www.utstat.utoronto.ca Delivered by Mark Ebden with thanks to Russ Salakhutdinov Outline Gaussian Processes Exam review Course evaluations
More informationLogistic Regression Review Fall 2012 Recitation. September 25, 2012 TA: Selen Uguroglu
Logistic Regression Review 10-601 Fall 2012 Recitation September 25, 2012 TA: Selen Uguroglu!1 Outline Decision Theory Logistic regression Goal Loss function Inference Gradient Descent!2 Training Data
More informationMachine Learning. 7. Logistic and Linear Regression
Sapienza University of Rome, Italy - Machine Learning (27/28) University of Rome La Sapienza Master in Artificial Intelligence and Robotics Machine Learning 7. Logistic and Linear Regression Luca Iocchi,
More informationECE 5984: Introduction to Machine Learning
ECE 5984: Introduction to Machine Learning Topics: Classification: Logistic Regression NB & LR connections Readings: Barber 17.4 Dhruv Batra Virginia Tech Administrativia HW2 Due: Friday 3/6, 3/15, 11:55pm
More informationLinear Regression (9/11/13)
STA561: Probabilistic machine learning Linear Regression (9/11/13) Lecturer: Barbara Engelhardt Scribes: Zachary Abzug, Mike Gloudemans, Zhuosheng Gu, Zhao Song 1 Why use linear regression? Figure 1: Scatter
More informationIterative Reweighted Least Squares
Iterative Reweighted Least Squares Sargur. University at Buffalo, State University of ew York USA Topics in Linear Classification using Probabilistic Discriminative Models Generative vs Discriminative
More informationBANA 7046 Data Mining I Lecture 4. Logistic Regression and Classications 1
BANA 7046 Data Mining I Lecture 4. Logistic Regression and Classications 1 Shaobo Li University of Cincinnati 1 Partially based on Hastie, et al. (2009) ESL, and James, et al. (2013) ISLR Data Mining I
More informationBias-Variance Tradeoff
What s learning, revisited Overfitting Generative versus Discriminative Logistic Regression Machine Learning 10701/15781 Carlos Guestrin Carnegie Mellon University September 19 th, 2007 Bias-Variance Tradeoff
More informationECE521 week 3: 23/26 January 2017
ECE521 week 3: 23/26 January 2017 Outline Probabilistic interpretation of linear regression - Maximum likelihood estimation (MLE) - Maximum a posteriori (MAP) estimation Bias-variance trade-off Linear
More informationHOMEWORK #4: LOGISTIC REGRESSION
HOMEWORK #4: LOGISTIC REGRESSION Probabilistic Learning: Theory and Algorithms CS 274A, Winter 2019 Due: 11am Monday, February 25th, 2019 Submit scan of plots/written responses to Gradebook; submit your
More informationGeneralized Linear Models. Last time: Background & motivation for moving beyond linear
Generalized Linear Models Last time: Background & motivation for moving beyond linear regression - non-normal/non-linear cases, binary, categorical data Today s class: 1. Examples of count and ordered
More informationRegression with Numerical Optimization. Logistic
CSG220 Machine Learning Fall 2008 Regression with Numerical Optimization. Logistic regression Regression with Numerical Optimization. Logistic regression based on a document by Andrew Ng October 3, 204
More informationMachine Learning Practice Page 2 of 2 10/28/13
Machine Learning 10-701 Practice Page 2 of 2 10/28/13 1. True or False Please give an explanation for your answer, this is worth 1 pt/question. (a) (2 points) No classifier can do better than a naive Bayes
More informationSupport Vector Machines
Support Vector Machines Le Song Machine Learning I CSE 6740, Fall 2013 Naïve Bayes classifier Still use Bayes decision rule for classification P y x = P x y P y P x But assume p x y = 1 is fully factorized
More informationσ(a) = a N (x; 0, 1 2 ) dx. σ(a) = Φ(a) =
Until now we have always worked with likelihoods and prior distributions that were conjugate to each other, allowing the computation of the posterior distribution to be done in closed form. Unfortunately,
More informationLinear Methods for Prediction
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike License. Your use of this material constitutes acceptance of that license and the conditions of use of materials on this
More informationLINEAR CLASSIFICATION, PERCEPTRON, LOGISTIC REGRESSION, SVC, NAÏVE BAYES. Supervised Learning
LINEAR CLASSIFICATION, PERCEPTRON, LOGISTIC REGRESSION, SVC, NAÏVE BAYES Supervised Learning Linear vs non linear classifiers In K-NN we saw an example of a non-linear classifier: the decision boundary
More informationMidterm Review CS 7301: Advanced Machine Learning. Vibhav Gogate The University of Texas at Dallas
Midterm Review CS 7301: Advanced Machine Learning Vibhav Gogate The University of Texas at Dallas Supervised Learning Issues in supervised learning What makes learning hard Point Estimation: MLE vs Bayesian
More informationPart 8: GLMs and Hierarchical LMs and GLMs
Part 8: GLMs and Hierarchical LMs and GLMs 1 Example: Song sparrow reproductive success Arcese et al., (1992) provide data on a sample from a population of 52 female song sparrows studied over the course
More informationLecture 5: Classification
Lecture 5: Classification Advanced Applied Multivariate Analysis STAT 2221, Spring 2015 Sungkyu Jung Department of Statistics, University of Pittsburgh Xingye Qiao Department of Mathematical Sciences Binghamton
More informationLearning with Noisy Labels. Kate Niehaus Reading group 11-Feb-2014
Learning with Noisy Labels Kate Niehaus Reading group 11-Feb-2014 Outline Motivations Generative model approach: Lawrence, N. & Scho lkopf, B. Estimating a Kernel Fisher Discriminant in the Presence of
More informationMIT Spring 2016
Generalized Linear Models MIT 18.655 Dr. Kempthorne Spring 2016 1 Outline Generalized Linear Models 1 Generalized Linear Models 2 Generalized Linear Model Data: (y i, x i ), i = 1,..., n where y i : response
More informationMidterm exam CS 189/289, Fall 2015
Midterm exam CS 189/289, Fall 2015 You have 80 minutes for the exam. Total 100 points: 1. True/False: 36 points (18 questions, 2 points each). 2. Multiple-choice questions: 24 points (8 questions, 3 points
More informationMachine Learning. Linear Models. Fabio Vandin October 10, 2017
Machine Learning Linear Models Fabio Vandin October 10, 2017 1 Linear Predictors and Affine Functions Consider X = R d Affine functions: L d = {h w,b : w R d, b R} where ( d ) h w,b (x) = w, x + b = w
More informationMachine Learning. Lecture 4: Regularization and Bayesian Statistics. Feng Li. https://funglee.github.io
Machine Learning Lecture 4: Regularization and Bayesian Statistics Feng Li fli@sdu.edu.cn https://funglee.github.io School of Computer Science and Technology Shandong University Fall 207 Overfitting Problem
More informationGeneralized Linear Models and Exponential Families
Generalized Linear Models and Exponential Families David M. Blei COS424 Princeton University April 12, 2012 Generalized Linear Models x n y n β Linear regression and logistic regression are both linear
More informationMLPR: Logistic Regression and Neural Networks
MLPR: Logistic Regression and Neural Networks Machine Learning and Pattern Recognition Amos Storkey Amos Storkey MLPR: Logistic Regression and Neural Networks 1/28 Outline 1 Logistic Regression 2 Multi-layer
More informationOutline. MLPR: Logistic Regression and Neural Networks Machine Learning and Pattern Recognition. Which is the correct model? Recap.
Outline MLPR: and Neural Networks Machine Learning and Pattern Recognition 2 Amos Storkey Amos Storkey MLPR: and Neural Networks /28 Recap Amos Storkey MLPR: and Neural Networks 2/28 Which is the correct
More informationHOMEWORK #4: LOGISTIC REGRESSION
HOMEWORK #4: LOGISTIC REGRESSION Probabilistic Learning: Theory and Algorithms CS 274A, Winter 2018 Due: Friday, February 23rd, 2018, 11:55 PM Submit code and report via EEE Dropbox You should submit a
More informationGenerative v. Discriminative classifiers Intuition
Logistic Regression (Continued) Generative v. Discriminative Decision rees Machine Learning 10701/15781 Carlos Guestrin Carnegie Mellon University January 31 st, 2007 2005-2007 Carlos Guestrin 1 Generative
More informationLecture 16 Solving GLMs via IRWLS
Lecture 16 Solving GLMs via IRWLS 09 November 2015 Taylor B. Arnold Yale Statistics STAT 312/612 Notes problem set 5 posted; due next class problem set 6, November 18th Goals for today fixed PCA example
More informationFinal Overview. Introduction to ML. Marek Petrik 4/25/2017
Final Overview Introduction to ML Marek Petrik 4/25/2017 This Course: Introduction to Machine Learning Build a foundation for practice and research in ML Basic machine learning concepts: max likelihood,
More informationCSC242: Intro to AI. Lecture 21
CSC242: Intro to AI Lecture 21 Administrivia Project 4 (homeworks 18 & 19) due Mon Apr 16 11:59PM Posters Apr 24 and 26 You need an idea! You need to present it nicely on 2-wide by 4-high landscape pages
More information