Logistic Regression. INFO-2301: Quantitative Reasoning 2 Michael Paul and Jordan Boyd-Graber SLIDES ADAPTED FROM HINRICH SCHÜTZE
|
|
|
- Myrtle James
- 5 years ago
- Views:
Transcription
1 Logistic Regression INFO-2301: Quantitative Reasoning 2 Michael Paul and Jordan Boyd-Graber SLIDES ADAPTED FROM HINRICH SCHÜTZE INFO-2301: Quantitative Reasoning 2 Paul and Boyd-Graber Logistic Regression 1 of 5
2 What are we talking about? Statistical classification: p(y x) y is typically a Bernoulli or multinomial outcome Classification uses: ad placement, spam detection Building block of other machine learning methods INFO-2301: Quantitative Reasoning 2 Paul and Boyd-Graber Logistic Regression 2 of 5
3 Logistic Regression: Definition Weight vector β i Observations X i Bias β 0 (like intercept in linear regression) 1 P(Y = 0 X) = 1 + exp β 0 + i β (1) ix i P(Y = 1 X) = exp β 0 + i β ix i 1 + exp β 0 + i β (2) ix i For shorthand, we ll say that P(Y = 0 X) = σ( (β 0 + β i X i )) (3) Where σ(z) = i P(Y = 1 X) = 1 σ( (β 0 + β i X i )) (4) 1 1+exp[ z] INFO-2301: Quantitative Reasoning 2 Paul and Boyd-Graber Logistic Regression 3 of 5 i
4 What s this exp doing? Exponential exp[x] is shorthand for e x e is a special number, about Logistic e x is the limit of compound interest formula as compounds become infinitely small It s the function whose derivative is itself The logistic function is σ(z) = 1 1+e z Looks like an S Always between 0 and 1. INFO-2301: Quantitative Reasoning 2 Paul and Boyd-Graber Logistic Regression 4 of 5
![What s this exp doing? Exponential exp[x] is shorthand for e x e is a special number, about 2.](/docs-images/87/96914441/images/5-2.jpg "71828 Logistic e x is the limit of compound interest formula as compounds become infinitely small It s the function whose derivative is itself The logistic function is σ(z) = 1 1+e z Looks like an S")
5 What s this exp doing? Exponential exp[x] is shorthand for e x e is a special number, about Logistic e x is the limit of compound interest formula as compounds become infinitely small It s the function whose derivative is itself The logistic function is σ(z) = 1 1+e z Looks like an S Always between 0 and 1. Allows us to model probabilities Different from linear regression INFO-2301: Quantitative Reasoning 2 Paul and Boyd-Graber Logistic Regression 4 of 5
6 Logistic Regression Example feature coefficient weight bias β viagra β mother β work β nigeria β Example 1: Empty Document? X = {} What does Y = 1 mean? INFO-2301: Quantitative Reasoning 2 Paul and Boyd-Graber Logistic Regression 5 of 5
7 Logistic Regression Example feature coefficient weight bias β viagra β mother β work β nigeria β Example 1: Empty Document? X = {} P(Y = 0) = P(Y = 1) = 1 1+exp[0.1] = exp[0.1] 1+exp[0.1] = What does Y = 1 mean? INFO-2301: Quantitative Reasoning 2 Paul and Boyd-Graber Logistic Regression 5 of 5
8 Logistic Regression Example feature coefficient weight bias β viagra β mother β work β nigeria β What does Y = 1 mean? Example 1: Empty Document? X = {} P(Y = 0) = 1 1+exp[0.1] = 0.48 P(Y = 1) = exp[0.1] 1+exp[0.1] = 0.52 Bias β 0 encodes the prior probability of a class INFO-2301: Quantitative Reasoning 2 Paul and Boyd-Graber Logistic Regression 5 of 5
9 Logistic Regression Example feature coefficient weight bias β viagra β mother β work β nigeria β Example 2 X = {Mother, Nigeria} What does Y = 1 mean? INFO-2301: Quantitative Reasoning 2 Paul and Boyd-Graber Logistic Regression 5 of 5
10 Logistic Regression Example feature coefficient weight bias β viagra β mother β work β nigeria β What does Y = 1 mean? Example 2 X = {Mother, Nigeria} P(Y = 0) = P(Y = 1) = 1 1+exp[ ] = exp[ ] 1+exp[ ] = Include bias, and sum the other weights INFO-2301: Quantitative Reasoning 2 Paul and Boyd-Graber Logistic Regression 5 of 5
11 Logistic Regression Example feature coefficient weight bias β viagra β mother β work β nigeria β What does Y = 1 mean? Example 2 X = {Mother, Nigeria} P(Y = 0) = 0.11 P(Y = 1) = exp[ ] = exp[ ] 1+exp[ ] = Include bias, and sum the other weights INFO-2301: Quantitative Reasoning 2 Paul and Boyd-Graber Logistic Regression 5 of 5
12 Logistic Regression Example feature coefficient weight bias β viagra β mother β work β nigeria β Example 3 X = {Mother, Work, Viagra, Mother} What does Y = 1 mean? INFO-2301: Quantitative Reasoning 2 Paul and Boyd-Graber Logistic Regression 5 of 5
13 Logistic Regression Example feature coefficient weight bias β viagra β mother β work β nigeria β What does Y = 1 mean? Example 3 X = {Mother, Work, Viagra, Mother} P(Y = 0) = 1 1+exp[ ] = P(Y = 1) = exp[ ] 1+exp[ ] = Multiply feature presence by weight INFO-2301: Quantitative Reasoning 2 Paul and Boyd-Graber Logistic Regression 5 of 5
14 Logistic Regression Example feature coefficient weight bias β viagra β mother β work β nigeria β What does Y = 1 mean? Example 3 X = {Mother, Work, Viagra, Mother} P(Y = 0) = 1 1+exp[ ] = 0.60 P(Y = 1) = exp[ ] 1+exp[ ] = 0.30 Multiply feature presence by weight INFO-2301: Quantitative Reasoning 2 Paul and Boyd-Graber Logistic Regression 5 of 5
15 Logistic Regression INFO-2301: Quantitative Reasoning 2 Michael Paul and Jordan Boyd-Graber ABC INFO-2301: Quantitative Reasoning 2 Paul and Boyd-Graber Logistic Regression 1 of 4
16 Logistic Regression: Objective Function l lnp(y X,β) = lnp(y (j) x (j),β) (1) j = y β (j) 0 + β i x (j) i ln 1 + exp β 0 + j i i β i x (j) i (2) INFO-2301: Quantitative Reasoning 2 Paul and Boyd-Graber Logistic Regression 2 of 4
17 Logistic Regression: Objective Function l lnp(y X,β) = lnp(y (j) x (j),β) (1) j = y β (j) 0 + β i x (j) i ln 1 + exp β 0 + j i i β i x (j) i Training data (y,x) are fixed. Objective function is a function of β... what values of β give a good value. (2) INFO-2301: Quantitative Reasoning 2 Paul and Boyd-Graber Logistic Regression 2 of 4
18 Convexity Convex function Doesn t matter where you start, if you walk up objective INFO-2301: Quantitative Reasoning 2 Paul and Boyd-Graber Logistic Regression 3 of 4
19 Convexity Convex function Doesn t matter where you start, if you walk up objective Gradient! INFO-2301: Quantitative Reasoning 2 Paul and Boyd-Graber Logistic Regression 3 of 4
20 Gradient Ascent (non-convex) Goal Optimize log likelihood with respect to variables β Objective Parameter INFO-2301: Quantitative Reasoning 2 Paul and Boyd-Graber Logistic Regression 4 of 4
21 Gradient Ascent (non-convex) Goal Optimize log likelihood with respect to variables β Objective Undiscovered Country Parameter INFO-2301: Quantitative Reasoning 2 Paul and Boyd-Graber Logistic Regression 4 of 4
22 Gradient Ascent (non-convex) Goal Optimize log likelihood with respect to variables β Objective Undiscovered Country 0 Parameter INFO-2301: Quantitative Reasoning 2 Paul and Boyd-Graber Logistic Regression 4 of 4
23 Gradient Ascent (non-convex) Goal Optimize log likelihood with respect to variables β Objective Undiscovered Country 0 Parameter INFO-2301: Quantitative Reasoning 2 Paul and Boyd-Graber Logistic Regression 4 of 4
24 Gradient Ascent (non-convex) Goal Optimize log likelihood with respect to variables β Objective Undiscovered Country 1 0 Parameter INFO-2301: Quantitative Reasoning 2 Paul and Boyd-Graber Logistic Regression 4 of 4
25 Gradient Ascent (non-convex) Goal Optimize log likelihood with respect to variables β Objective Undiscovered Country 1 0 Parameter INFO-2301: Quantitative Reasoning 2 Paul and Boyd-Graber Logistic Regression 4 of 4
26 Gradient Ascent (non-convex) Goal Optimize log likelihood with respect to variables β Objective Undiscovered Country Parameter INFO-2301: Quantitative Reasoning 2 Paul and Boyd-Graber Logistic Regression 4 of 4
27 Gradient Ascent (non-convex) Goal Optimize log likelihood with respect to variables β Objective Undiscovered Country Parameter INFO-2301: Quantitative Reasoning 2 Paul and Boyd-Graber Logistic Regression 4 of 4
28 Gradient Ascent (non-convex) Goal Optimize log likelihood with respect to variables β Objective Undiscovered Country Parameter INFO-2301: Quantitative Reasoning 2 Paul and Boyd-Graber Logistic Regression 4 of 4
29 Gradient Ascent (non-convex) Goal Optimize log likelihood with respect to variables β Objective Undiscovered Country Parameter Luckily, (vanilla) logistic regression is convex INFO-2301: Quantitative Reasoning 2 Paul and Boyd-Graber Logistic Regression 4 of 4
30 Logistic Regression INFO-2301: Quantitative Reasoning 2 Michael Paul and Jordan Boyd-Graber SLIDES ADAPTED FROM WILLIAM COHEN INFO-2301: Quantitative Reasoning 2 Paul and Boyd-Graber Logistic Regression 1 of 10
31 Gradient for Logistic Regression To ease notation, let s define π i = expβt x i 1 + expβ T x i (1) Our objective function is logπ i if y i = 1 l = logp(y i x i ) = l i = log(1 π i ) if y i = 0 i i i (2) INFO-2301: Quantitative Reasoning 2 Paul and Boyd-Graber Logistic Regression 2 of 10
32 Taking the Derivative Apply chain rule: l β j = i l i ( β) = β j i 1 πi π i β if y j i = π i π i β if y j i = 0 (3) If we plug in the derivative, we can merge these two cases π i β j = π i (1 π i )x j, (4) l i β j = (y i π i )x j. (5) INFO-2301: Quantitative Reasoning 2 Paul and Boyd-Graber Logistic Regression 3 of 10
33 Gradient for Logistic Regression Gradient β l( β) = l( β) β 0,..., l( β) β n (6) Update β η β l( β) (7) β i β i + η l( β) β i (8) INFO-2301: Quantitative Reasoning 2 Paul and Boyd-Graber Logistic Regression 4 of 10
34 Gradient for Logistic Regression Gradient β l( β) = l( β) β 0,..., l( β) β n (6) Update β η β l( β) (7) β i β i + η l( β) β i (8) Why are we adding? What would well do if we wanted to do descent? INFO-2301: Quantitative Reasoning 2 Paul and Boyd-Graber Logistic Regression 4 of 10
35 Gradient for Logistic Regression Gradient β l( β) = l( β) β 0,..., l( β) β n (6) Update η: step size, must be greater than zero β η β l( β) (7) β i β i + η l( β) β i (8) INFO-2301: Quantitative Reasoning 2 Paul and Boyd-Graber Logistic Regression 4 of 10
36 Gradient for Logistic Regression Gradient β l( β) = l( β) β 0,..., l( β) β n (6) Update β η β l( β) (7) β i β i + η l( β) β i (8) NB: Conjugate gradient is usually better, but harder to implement INFO-2301: Quantitative Reasoning 2 Paul and Boyd-Graber Logistic Regression 4 of 10
37 Choosing Step Size Objective Parameter INFO-2301: Quantitative Reasoning 2 Paul and Boyd-Graber Logistic Regression 5 of 10
38 Choosing Step Size Objective Parameter INFO-2301: Quantitative Reasoning 2 Paul and Boyd-Graber Logistic Regression 5 of 10
39 Choosing Step Size Objective Parameter INFO-2301: Quantitative Reasoning 2 Paul and Boyd-Graber Logistic Regression 5 of 10
40 Choosing Step Size Objective Parameter INFO-2301: Quantitative Reasoning 2 Paul and Boyd-Graber Logistic Regression 5 of 10
41 Choosing Step Size Objective Parameter INFO-2301: Quantitative Reasoning 2 Paul and Boyd-Graber Logistic Regression 5 of 10
42 Approximating the Gradient Our datasets are big (to fit into memory)... or data are changing / streaming INFO-2301: Quantitative Reasoning 2 Paul and Boyd-Graber Logistic Regression 6 of 10
43 Approximating the Gradient Our datasets are big (to fit into memory)... or data are changing / streaming Hard to compute true gradient l(β) x [ l(β,x)] (9) Average over all observations INFO-2301: Quantitative Reasoning 2 Paul and Boyd-Graber Logistic Regression 6 of 10
44 Approximating the Gradient Our datasets are big (to fit into memory)... or data are changing / streaming Hard to compute true gradient l(β) x [ l(β,x)] (9) Average over all observations What if we compute an update just from one observation? INFO-2301: Quantitative Reasoning 2 Paul and Boyd-Graber Logistic Regression 6 of 10
45 Getting to Union Station Pretend it s a pre-smartphone world and you want to get to Union Station INFO-2301: Quantitative Reasoning 2 Paul and Boyd-Graber Logistic Regression 7 of 10
46 Stochastic Gradient for Logistic Regression Given a single observation x i chosen at random from the dataset, β j β j + η[y i π i ]x i,j (10) INFO-2301: Quantitative Reasoning 2 Paul and Boyd-Graber Logistic Regression 8 of 10
47 Stochastic Gradient for Logistic Regression Given a single observation x i chosen at random from the dataset, Examples in class. β j β j + η[y i π i ]x i,j (10) INFO-2301: Quantitative Reasoning 2 Paul and Boyd-Graber Logistic Regression 8 of 10
48 Algorithm 1 Initialize a vector B to be all zeros 2 For t = 1,...,T For each example x i,y i and feature j: Compute π i Pr(y i = 1 x i ) Set β[j] = β[j] + λ(y i π i )x i 3 Output the parameters β 1,...,β d. INFO-2301: Quantitative Reasoning 2 Paul and Boyd-Graber Logistic Regression 9 of 10
49 Wrapup Logistic Regression: Regression for outputting Probabilities Intuitions similar to linear regression We ll talk about feature engineering for both next time INFO-2301: Quantitative Reasoning 2 Paul and Boyd-Graber Logistic Regression 10 of 10
Introduction to Machine Learning
Introduction to Machine Learning Machine Learning: Jordan Boyd-Graber University of Maryland LOGISTIC REGRESSION FROM TEXT Slides adapted from Emily Fox Machine Learning: Jordan Boyd-Graber UMD Introduction
Classification: Logistic Regression from Data
Classification: Logistic Regression from Data Machine Learning: Alvin Grissom II University of Colorado Boulder Slides adapted from Emily Fox Machine Learning: Alvin Grissom II Boulder Classification:
Logistic Regression. Introduction to Data Science Algorithms Jordan Boyd-Graber and Michael Paul SLIDES ADAPTED FROM HINRICH SCHÜTZE
Logistic Regression Introduction to Data Science Algorithms Jordan Boyd-Graber and Michael Paul SLIDES ADAPTED FROM HINRICH SCHÜTZE Introduction to Data Science Algorithms Boyd-Graber and Paul Logistic
Classification: Logistic Regression from Data
Classification: Logistic Regression from Data Machine Learning: Jordan Boyd-Graber University of Colorado Boulder LECTURE 3 Slides adapted from Emily Fox Machine Learning: Jordan Boyd-Graber Boulder Classification:
Machine Learning: Chenhao Tan University of Colorado Boulder LECTURE 5
Machine Learning: Chenhao Tan University of Colorado Boulder LECTURE 5 Slides adapted from Jordan Boyd-Graber, Tom Mitchell, Ziv Bar-Joseph Machine Learning: Chenhao Tan Boulder 1 of 27 Quiz question For
Variations of Logistic Regression with Stochastic Gradient Descent
Variations of Logistic Regression with Stochastic Gradient Descent Panqu Wang(pawang@ucsd.edu) Phuc Xuan Nguyen(pxn002@ucsd.edu) January 26, 2012 Abstract In this paper, we extend the traditional logistic
Optimization and Gradient Descent
Optimization and Gradient Descent INFO-4604, Applied Machine Learning University of Colorado Boulder September 12, 2017 Prof. Michael Paul Prediction Functions Remember: a prediction function is the function
Outline. Supervised Learning. Hong Chang. Institute of Computing Technology, Chinese Academy of Sciences. Machine Learning Methods (Fall 2012)
Outline Hong Chang Institute of Computing Technology, Chinese Academy of Sciences Machine Learning Methods (Fall 2012) Outline Outline I 1 Linear Models for Regression Linear Regression Probabilistic Interpretation
Clustering. Introduction to Data Science. Jordan Boyd-Graber and Michael Paul SLIDES ADAPTED FROM LAUREN HANNAH
Clustering Introduction to Data Science Jordan Boyd-Graber and Michael Paul SLIDES ADAPTED FROM LAUREN HANNAH Slides adapted from Tom Mitchell, Eric Xing, and Lauren Hannah Introduction to Data Science
Logistic Regression. William Cohen
Logistic Regression William Cohen 1 Outline Quick review classi5ication, naïve Bayes, perceptrons new result for naïve Bayes Learning as optimization Logistic regression via gradient ascent Over5itting
Logistic Regression: Online, Lazy, Kernelized, Sequential, etc.
Logistic Regression: Online, Lazy, Kernelized, Sequential, etc. Harsha Veeramachaneni Thomson Reuter Research and Development April 1, 2010 Harsha Veeramachaneni (TR R&D) Logistic Regression April 1, 2010
Stochastic Gradient Descent
Stochastic Gradient Descent Machine Learning CSE546 Carlos Guestrin University of Washington October 9, 2013 1 Logistic Regression Logistic function (or Sigmoid): Learn P(Y X) directly Assume a particular
Logistic Regression. COMP 527 Danushka Bollegala
Logistic Regression COMP 527 Danushka Bollegala Binary Classification Given an instance x we must classify it to either positive (1) or negative (0) class We can use {1,-1} instead of {1,0} but we will
Classification CE-717: Machine Learning Sharif University of Technology. M. Soleymani Fall 2012
Classification CE-717: Machine Learning Sharif University of Technology M. Soleymani Fall 2012 Topics Discriminant functions Logistic regression Perceptron Generative models Generative vs. discriminative
Machine Learning. Regression-Based Classification & Gaussian Discriminant Analysis. Manfred Huber
Machine Learning Regression-Based Classification & Gaussian Discriminant Analysis Manfred Huber 2015 1 Logistic Regression Linear regression provides a nice representation and an efficient solution to
Maximum Likelihood, Logistic Regression, and Stochastic Gradient Training
Maximum Likelihood, Logistic Regression, and Stochastic Gradient Training Charles Elkan elkan@cs.ucsd.edu January 17, 2013 1 Principle of maximum likelihood Consider a family of probability distributions
Logistic Regression Logistic
Case Study 1: Estimating Click Probabilities L2 Regularization for Logistic Regression Machine Learning/Statistics for Big Data CSE599C1/STAT592, University of Washington Carlos Guestrin January 10 th,
Solving Regression. Jordan Boyd-Graber. University of Colorado Boulder LECTURE 12. Slides adapted from Matt Nedrich and Trevor Hastie
Solving Regression Jordan Boyd-Graber University of Colorado Boulder LECTURE 12 Slides adapted from Matt Nedrich and Trevor Hastie Jordan Boyd-Graber Boulder Solving Regression 1 of 17 Roadmap We talked
Introduction to Machine Learning. Regression. Computer Science, Tel-Aviv University,
1 Introduction to Machine Learning Regression Computer Science, Tel-Aviv University, 2013-14 Classification Input: X Real valued, vectors over real. Discrete values (0,1,2,...) Other structures (e.g.,
Midterm. Introduction to Machine Learning. CS 189 Spring You have 1 hour 20 minutes for the exam.
CS 189 Spring 2013 Introduction to Machine Learning Midterm You have 1 hour 20 minutes for the exam. The exam is closed book, closed notes except your one-page crib sheet. Please use non-programmable calculators
Case Study 1: Estimating Click Probabilities. Kakade Announcements: Project Proposals: due this Friday!
Case Study 1: Estimating Click Probabilities Intro Logistic Regression Gradient Descent + SGD Machine Learning for Big Data CSE547/STAT548, University of Washington Sham Kakade April 4, 017 1 Announcements:
Support Vector Machines
Support Vector Machines Jordan Boyd-Graber University of Colorado Boulder LECTURE 7 Slides adapted from Tom Mitchell, Eric Xing, and Lauren Hannah Jordan Boyd-Graber Boulder Support Vector Machines 1 of
Regression with Numerical Optimization. Logistic
CSG220 Machine Learning Fall 2008 Regression with Numerical Optimization. Logistic regression Regression with Numerical Optimization. Logistic regression based on a document by Andrew Ng October 3, 204
Generative v. Discriminative classifiers Intuition
Logistic Regression Machine Learning 10701/15781 Carlos Guestrin Carnegie Mellon University September 24 th, 2007 1 Generative v. Discriminative classifiers Intuition Want to Learn: h:x a Y X features
Ad Placement Strategies
Case Study : Estimating Click Probabilities Intro Logistic Regression Gradient Descent + SGD AdaGrad Machine Learning for Big Data CSE547/STAT548, University of Washington Emily Fox January 7 th, 04 Ad
CSE 417T: Introduction to Machine Learning. Lecture 11: Review. Henry Chai 10/02/18
CSE 417T: Introduction to Machine Learning Lecture 11: Review Henry Chai 10/02/18 Unknown Target Function!: # % Training data Formal Setup & = ( ), + ),, ( -, + - Learning Algorithm 2 Hypothesis Set H
Generative v. Discriminative classifiers Intuition
Logistic Regression Machine Learning 070/578 Carlos Guestrin Carnegie Mellon University September 24 th, 2007 Generative v. Discriminative classifiers Intuition Want to Learn: h:x a Y X features Y target
Machine Learning. Lecture 3: Logistic Regression. Feng Li.
Machine Learning Lecture 3: Logistic Regression Feng Li fli@sdu.edu.cn https://funglee.github.io School of Computer Science and Technology Shandong University Fall 2016 Logistic Regression Classification
Introduction to Machine Learning
Introduction to Machine Learning Logistic Regression Varun Chandola Computer Science & Engineering State University of New York at Buffalo Buffalo, NY, USA chandola@buffalo.edu Chandola@UB CSE 474/574
Bias-Variance Tradeoff
What s learning, revisited Overfitting Generative versus Discriminative Logistic Regression Machine Learning 10701/15781 Carlos Guestrin Carnegie Mellon University September 19 th, 2007 Bias-Variance Tradeoff
Overfitting, Bias / Variance Analysis
Overfitting, Bias / Variance Analysis Professor Ameet Talwalkar Professor Ameet Talwalkar CS260 Machine Learning Algorithms February 8, 207 / 40 Outline Administration 2 Review of last lecture 3 Basic
Logistic Regression. Will Monroe CS 109. Lecture Notes #22 August 14, 2017
1 Will Monroe CS 109 Logistic Regression Lecture Notes #22 August 14, 2017 Based on a chapter by Chris Piech Logistic regression is a classification algorithm1 that works by trying to learn a function
Lecture 2: Logistic Regression and Neural Networks
1/23 Lecture 2: and Neural Networks Pedro Savarese TTI 2018 2/23 Table of Contents 1 2 3 4 3/23 Naive Bayes Learn p(x, y) = p(y)p(x y) Training: Maximum Likelihood Estimation Issues? Why learn p(x, y)
Introduction to Machine Learning
Introduction to Machine Learning Machine Learning: Jordan Boyd-Graber University of Maryland SUPPORT VECTOR MACHINES Slides adapted from Tom Mitchell, Eric Xing, and Lauren Hannah Machine Learning: Jordan
MLE/MAP + Naïve Bayes
10-601 Introduction to Machine Learning Machine Learning Department School of Computer Science Carnegie Mellon University MLE/MAP + Naïve Bayes MLE / MAP Readings: Estimating Probabilities (Mitchell, 2016)
Logistic Regression. Advanced Methods for Data Analysis (36-402/36-608) Spring 2014
Logistic Regression Advanced Methods for Data Analysis (36-402/36-608 Spring 204 Classification. Introduction to classification Classification, like regression, is a predictive task, but one in which the
Introduction to Logistic Regression and Support Vector Machine
Introduction to Logistic Regression and Support Vector Machine guest lecturer: Ming-Wei Chang CS 446 Fall, 2009 () / 25 Fall, 2009 / 25 Before we start () 2 / 25 Fall, 2009 2 / 25 Before we start Feel
Logistic Regression Introduction to Machine Learning. Matt Gormley Lecture 9 Sep. 26, 2018
10-601 Introduction to Machine Learning Machine Learning Department School of Computer Science Carnegie Mellon University Logistic Regression Matt Gormley Lecture 9 Sep. 26, 2018 1 Reminders Homework 3:
Machine Learning Tom M. Mitchell Machine Learning Department Carnegie Mellon University. September 20, 2012
Machine Learning 10-601 Tom M. Mitchell Machine Learning Department Carnegie Mellon University September 20, 2012 Today: Logistic regression Generative/Discriminative classifiers Readings: (see class website)
Machine Learning
Machine Learning 10-601 Tom M. Mitchell Machine Learning Department Carnegie Mellon University February 2, 2015 Today: Logistic regression Generative/Discriminative classifiers Readings: (see class website)
Inexact Search is Good Enough
Inexact Search is Good Enough Advanced Machine Learning for NLP Jordan Boyd-Graber MATHEMATICAL TREATMENT Advanced Machine Learning for NLP Boyd-Graber Inexact Search is Good Enough 1 of 1 Preliminaries:
Machine Learning Basics Lecture 2: Linear Classification. Princeton University COS 495 Instructor: Yingyu Liang
Machine Learning Basics Lecture 2: Linear Classification Princeton University COS 495 Instructor: Yingyu Liang Review: machine learning basics Math formulation Given training data x i, y i : 1 i n i.i.d.
CPSC 340: Machine Learning and Data Mining. Stochastic Gradient Fall 2017
CPSC 340: Machine Learning and Data Mining Stochastic Gradient Fall 2017 Assignment 3: Admin Check update thread on Piazza for correct definition of trainndx. This could make your cross-validation code
Week 5: Logistic Regression & Neural Networks
Week 5: Logistic Regression & Neural Networks Instructor: Sergey Levine 1 Summary: Logistic Regression In the previous lecture, we covered logistic regression. To recap, logistic regression models and
Logistic Regression Trained with Different Loss Functions. Discussion
Logistic Regression Trained with Different Loss Functions Discussion CS640 Notations We restrict our discussions to the binary case. g(z) = g (z) = g(z) z h w (x) = g(wx) = + e z = g(z)( g(z)) + e wx =
Linear Models in Machine Learning
CS540 Intro to AI Linear Models in Machine Learning Lecturer: Xiaojin Zhu jerryzhu@cs.wisc.edu We briefly go over two linear models frequently used in machine learning: linear regression for, well, regression,
Generative v. Discriminative classifiers Intuition
Logistic Regression (Continued) Generative v. Discriminative Decision rees Machine Learning 10701/15781 Carlos Guestrin Carnegie Mellon University January 31 st, 2007 2005-2007 Carlos Guestrin 1 Generative
Support Vector Machines
Support Vector Machines Le Song Machine Learning I CSE 6740, Fall 2013 Naïve Bayes classifier Still use Bayes decision rule for classification P y x = P x y P y P x But assume p x y = 1 is fully factorized
Linear Models for Classification
Linear Models for Classification Oliver Schulte - CMPT 726 Bishop PRML Ch. 4 Classification: Hand-written Digit Recognition CHINE INTELLIGENCE, VOL. 24, NO. 24, APRIL 2002 x i = t i = (0, 0, 0, 1, 0, 0,
Linear Models for Regression
Linear Models for Regression CSE 4309 Machine Learning Vassilis Athitsos Computer Science and Engineering Department University of Texas at Arlington 1 The Regression Problem Training data: A set of input-output
Gaussian and Linear Discriminant Analysis; Multiclass Classification
Gaussian and Linear Discriminant Analysis; Multiclass Classification Professor Ameet Talwalkar Slide Credit: Professor Fei Sha Professor Ameet Talwalkar CS260 Machine Learning Algorithms October 13, 2015
Neural Networks: Backpropagation
Neural Networks: Backpropagation Seung-Hoon Na 1 1 Department of Computer Science Chonbuk National University 2018.10.25 eung-hoon Na (Chonbuk National University) Neural Networks: Backpropagation 2018.10.25
CS229 Supplemental Lecture notes
CS229 Supplemental Lecture notes John Duchi Binary classification In binary classification problems, the target y can take on at only two values. In this set of notes, we show how to model this problem
Sequence Modelling with Features: Linear-Chain Conditional Random Fields. COMP-599 Oct 6, 2015
Sequence Modelling with Features: Linear-Chain Conditional Random Fields COMP-599 Oct 6, 2015 Announcement A2 is out. Due Oct 20 at 1pm. 2 Outline Hidden Markov models: shortcomings Generative vs. discriminative
Logistic Regression. Stochastic Gradient Descent
Tutorial 8 CPSC 340 Logistic Regression Stochastic Gradient Descent Logistic Regression Model A discriminative probabilistic model for classification e.g. spam filtering Let x R d be input and y { 1, 1}
Logistic Regression. Robot Image Credit: Viktoriya Sukhanova 123RF.com
Logistic Regression These slides were assembled by Eric Eaton, with grateful acknowledgement of the many others who made their course materials freely available online. Feel free to reuse or adapt these
Machine Learning, Fall 2012 Homework 2
0-60 Machine Learning, Fall 202 Homework 2 Instructors: Tom Mitchell, Ziv Bar-Joseph TA in charge: Selen Uguroglu email: sugurogl@cs.cmu.edu SOLUTIONS Naive Bayes, 20 points Problem. Basic concepts, 0
Logistic Regression & Neural Networks
Logistic Regression & Neural Networks CMSC 723 / LING 723 / INST 725 Marine Carpuat Slides credit: Graham Neubig, Jacob Eisenstein Logistic Regression Perceptron & Probabilities What if we want a probability
Generative Clustering, Topic Modeling, & Bayesian Inference
Generative Clustering, Topic Modeling, & Bayesian Inference INFO-4604, Applied Machine Learning University of Colorado Boulder December 12-14, 2017 Prof. Michael Paul Unsupervised Naïve Bayes Last week
Generative Learning. INFO-4604, Applied Machine Learning University of Colorado Boulder. November 29, 2018 Prof. Michael Paul
Generative Learning INFO-4604, Applied Machine Learning University of Colorado Boulder November 29, 2018 Prof. Michael Paul Generative vs Discriminative The classification algorithms we have seen so far
Probabilistic modeling. The slides are closely adapted from Subhransu Maji s slides
Probabilistic modeling The slides are closely adapted from Subhransu Maji s slides Overview So far the models and algorithms you have learned about are relatively disconnected Probabilistic modeling framework
Least Mean Squares Regression
Least Mean Squares Regression Machine Learning Spring 2018 The slides are mainly from Vivek Srikumar 1 Lecture Overview Linear classifiers What functions do linear classifiers express? Least Squares Method
Naïve Bayes Introduction to Machine Learning. Matt Gormley Lecture 18 Oct. 31, 2018
10-601 Introduction to Machine Learning Machine Learning Department School of Computer Science Carnegie Mellon University Naïve Bayes Matt Gormley Lecture 18 Oct. 31, 2018 1 Reminders Homework 6: PAC Learning
HOMEWORK #4: LOGISTIC REGRESSION
HOMEWORK #4: LOGISTIC REGRESSION Probabilistic Learning: Theory and Algorithms CS 274A, Winter 2018 Due: Friday, February 23rd, 2018, 11:55 PM Submit code and report via EEE Dropbox You should submit a
Logistic Regression Introduction to Machine Learning. Matt Gormley Lecture 8 Feb. 12, 2018
10-601 Introduction to Machine Learning Machine Learning Department School of Computer Science Carnegie Mellon University Logistic Regression Matt Gormley Lecture 8 Feb. 12, 2018 1 10-601 Introduction
Topic 2: Logistic Regression
CS 4850/6850: Introduction to Machine Learning Fall 208 Topic 2: Logistic Regression Instructor: Daniel L. Pimentel-Alarcón c Copyright 208 2. Introduction Arguably the simplest task that we can teach
Perceptron (Theory) + Linear Regression
10601 Introduction to Machine Learning Machine Learning Department School of Computer Science Carnegie Mellon University Perceptron (Theory) Linear Regression Matt Gormley Lecture 6 Feb. 5, 2018 1 Q&A
Probability Distributions: Continuous
Probability Distributions: Continuous INFO-2301: Quantitative Reasoning 2 Michael Paul and Jordan Boyd-Graber FEBRUARY 28, 2017 INFO-2301: Quantitative Reasoning 2 Paul and Boyd-Graber Probability Distributions:
Comments. x > w = w > x. Clarification: this course is about getting you to be able to think as a machine learning expert
Logistic regression Comments Mini-review and feedback These are equivalent: x > w = w > x Clarification: this course is about getting you to be able to think as a machine learning expert There has to be
Lecture 2 Machine Learning Review
Lecture 2 Machine Learning Review CMSC 35246: Deep Learning Shubhendu Trivedi & Risi Kondor University of Chicago March 29, 2017 Things we will look at today Formal Setup for Supervised Learning Things
Machine Learning Basics: Stochastic Gradient Descent. Sargur N. Srihari
Machine Learning Basics: Stochastic Gradient Descent Sargur N. srihari@cedar.buffalo.edu 1 Topics 1. Learning Algorithms 2. Capacity, Overfitting and Underfitting 3. Hyperparameters and Validation Sets
Generative Learning algorithms
CS9 Lecture notes Andrew Ng Part IV Generative Learning algorithms So far, we ve mainly been talking about learning algorithms that model p(y x; θ), the conditional distribution of y given x. For instance,
Machine Learning Gaussian Naïve Bayes Big Picture
Machine Learning 10-701 Tom M. Mitchell Machine Learning Department Carnegie Mellon University January 27, 2011 Today: Naïve Bayes Big Picture Logistic regression Gradient ascent Generative discriminative
Logistic Regression. Mohammad Emtiyaz Khan EPFL Oct 8, 2015
Logistic Regression Mohammad Emtiyaz Khan EPFL Oct 8, 2015 Mohammad Emtiyaz Khan 2015 Classification with linear regression We can use y = 0 for C 1 and y = 1 for C 2 (or vice-versa), and simply use least-squares
Lecture 3 - Linear and Logistic Regression
3 - Linear and Logistic Regression-1 Machine Learning Course Lecture 3 - Linear and Logistic Regression Lecturer: Haim Permuter Scribe: Ziv Aharoni Throughout this lecture we talk about how to use regression
Machine Learning: Chenhao Tan University of Colorado Boulder LECTURE 9
Machine Learning: Chenhao Tan University of Colorado Boulder LECTURE 9 Slides adapted from Jordan Boyd-Graber Machine Learning: Chenhao Tan Boulder 1 of 39 Recap Supervised learning Previously: KNN, naïve
Last Time. Today. Bayesian Learning. The Distributions We Love. CSE 446 Gaussian Naïve Bayes & Logistic Regression
CSE 446 Gaussian Naïve Bayes & Logistic Regression Winter 22 Dan Weld Learning Gaussians Naïve Bayes Last Time Gaussians Naïve Bayes Logistic Regression Today Some slides from Carlos Guestrin, Luke Zettlemoyer
Linear Regression (continued)
Linear Regression (continued) Professor Ameet Talwalkar Professor Ameet Talwalkar CS260 Machine Learning Algorithms February 6, 2017 1 / 39 Outline 1 Administration 2 Review of last lecture 3 Linear regression
Computational statistics
Computational statistics Lecture 3: Neural networks Thierry Denœux 5 March, 2016 Neural networks A class of learning methods that was developed separately in different fields statistics and artificial
Probabilistic classification CE-717: Machine Learning Sharif University of Technology. M. Soleymani Fall 2016
Probabilistic classification CE-717: Machine Learning Sharif University of Technology M. Soleymani Fall 2016 Topics Probabilistic approach Bayes decision theory Generative models Gaussian Bayes classifier
Neural Network Training
Neural Network Training Sargur Srihari Topics in Network Training 0. Neural network parameters Probabilistic problem formulation Specifying the activation and error functions for Regression Binary classification
Logistic Regression. Jia-Bin Huang. Virginia Tech Spring 2019 ECE-5424G / CS-5824
Logistic Regression Jia-Bin Huang ECE-5424G / CS-5824 Virginia Tech Spring 2019 Administrative Please start HW 1 early! Questions are welcome! Two principles for estimating parameters Maximum Likelihood
Linear Classification. CSE 6363 Machine Learning Vassilis Athitsos Computer Science and Engineering Department University of Texas at Arlington
Linear Classification CSE 6363 Machine Learning Vassilis Athitsos Computer Science and Engineering Department University of Texas at Arlington 1 Example of Linear Classification Red points: patterns belonging
Topic 3: Neural Networks
CS 4850/6850: Introduction to Machine Learning Fall 2018 Topic 3: Neural Networks Instructor: Daniel L. Pimentel-Alarcón c Copyright 2018 3.1 Introduction Neural networks are arguably the main reason why
Warm up: risk prediction with logistic regression
Warm up: risk prediction with logistic regression Boss gives you a bunch of data on loans defaulting or not: {(x i,y i )} n i= x i 2 R d, y i 2 {, } You model the data as: P (Y = y x, w) = + exp( yw T
Lecture 7. Logistic Regression. Luigi Freda. ALCOR Lab DIAG University of Rome La Sapienza. December 11, 2016
Lecture 7 Logistic Regression Luigi Freda ALCOR Lab DIAG University of Rome La Sapienza December 11, 2016 Luigi Freda ( La Sapienza University) Lecture 7 December 11, 2016 1 / 39 Outline 1 Intro Logistic
1 Review of Winnow Algorithm
COS 511: Theoretical Machine Learning Lecturer: Rob Schapire Lecture # 17 Scribe: Xingyuan Fang, Ethan April 9th, 2013 1 Review of Winnow Algorithm We have studied Winnow algorithm in Algorithm 1. Algorithm
Logistic Regression. Seungjin Choi
Logistic Regression Seungjin Choi Department of Computer Science and Engineering Pohang University of Science and Technology 77 Cheongam-ro, Nam-gu, Pohang 37673, Korea seungjin@postech.ac.kr http://mlg.postech.ac.kr/
HOMEWORK #4: LOGISTIC REGRESSION
HOMEWORK #4: LOGISTIC REGRESSION Probabilistic Learning: Theory and Algorithms CS 274A, Winter 2019 Due: 11am Monday, February 25th, 2019 Submit scan of plots/written responses to Gradebook; submit your
Statistical Machine Learning Hilary Term 2018
Statistical Machine Learning Hilary Term 2018 Pier Francesco Palamara Department of Statistics University of Oxford Slide credits and other course material can be found at: http://www.stats.ox.ac.uk/~palamara/sml18.html
Logistic Regression Review Fall 2012 Recitation. September 25, 2012 TA: Selen Uguroglu
Logistic Regression Review 10-601 Fall 2012 Recitation September 25, 2012 TA: Selen Uguroglu!1 Outline Decision Theory Logistic regression Goal Loss function Inference Gradient Descent!2 Training Data
Machine Learning (CSE 446): Multi-Class Classification; Kernel Methods
Machine Learning (CSE 446): Multi-Class Classification; Kernel Methods Sham M Kakade c 2018 University of Washington cse446-staff@cs.washington.edu 1 / 12 Announcements HW3 due date as posted. make sure
Ad Placement Strategies
Case Study 1: Estimating Click Probabilities Tackling an Unknown Number of Features with Sketching Machine Learning for Big Data CSE547/STAT548, University of Washington Emily Fox 2014 Emily Fox January
Linear & nonlinear classifiers
Linear & nonlinear classifiers Machine Learning Hamid Beigy Sharif University of Technology Fall 1394 Hamid Beigy (Sharif University of Technology) Linear & nonlinear classifiers Fall 1394 1 / 34 Table
Introduction to Logistic Regression
Introduction to Logistic Regression Guy Lebanon Binary Classification Binary classification is the most basic task in machine learning, and yet the most frequent. Binary classifiers often serve as the
Naïve Bayes classification
Naïve Bayes classification 1 Probability theory Random variable: a variable whose possible values are numerical outcomes of a random phenomenon. Examples: A person s height, the outcome of a coin toss
Machine Learning 4771
Machine Learning 4771 Instructor: Tony Jebara Topic 7 Unsupervised Learning Statistical Perspective Probability Models Discrete & Continuous: Gaussian, Bernoulli, Multinomial Maimum Likelihood Logistic
Lecture 17: Neural Networks and Deep Learning
UVA CS 6316 / CS 4501-004 Machine Learning Fall 2016 Lecture 17: Neural Networks and Deep Learning Jack Lanchantin Dr. Yanjun Qi 1 Neurons 1-Layer Neural Network Multi-layer Neural Network Loss Functions
Deep Feedforward Networks. Seung-Hoon Na Chonbuk National University
Deep Feedforward Networks Seung-Hoon Na Chonbuk National University Neural Network: Types Feedforward neural networks (FNN) = Deep feedforward networks = multilayer perceptrons (MLP) No feedback connections
Stochastic gradient descent; Classification
Stochastic gradient descent; Classification Steve Renals Machine Learning Practical MLP Lecture 2 28 September 2016 MLP Lecture 2 Stochastic gradient descent; Classification 1 Single Layer Networks MLP