CSE 151 Machine Learning. Instructor: Kamalika Chaudhuri
|
|
- Loreen Hines
- 6 years ago
- Views:
Transcription
1 CSE 151 Machine Learning Instructor: Kamalika Chaudhuri
2 Linear Classification Given labeled data: (xi, feature vector yi) label i=1,..,n where y is 1 or 1, find a hyperplane to separate from
3 Linear Classification Process Classification O Algorithm w Training Data Prediction Rule: Vector w How do you classify a test example x? Output y = sign(w, x)
4 Linear Classification Given labeled data (xi, yi), i=1,..,n, where y is 1 or 1, Find a hyperplane through the origin to separate from O w w: normal vector to the hyperplane For a point x on one side of the hyperplane, w, x > 0 For a point x on the other side, w, x < 0
5 The Perceptron Algorithm Goal: Given labeled data (xi, yi), i=1,..,n, where y is 1 or 1, Find a vector w such that the corresponding hyperplane separates from Perceptron Algorithm: 1. Initially, w1 = y1x1 2. For t=2, 3,... If y t w t,x t < 0 then: Else: wt1 = wt yt xt wt1 = wt
6 Linear Separability linearly separable not linearly separable
7 Measure of Separability: Margin For a unit vector w, and training set S, margin of w is defined as: γ =min x S w, x x min cosine of angle between w and x, for any x in S O θ w O θ w Low Margin High Margin
8 Real Examples: Margin For a unit vector w, and training set S, margin of w is defined as: γ =min x S w, x x min cosine of angle between w and x, for any x in data w w Low Margin High Margin
9 When does Perceptron work? Theorem: If the training points are linearly separable with margin γ and if x i =1 for all xi in the training data, then #mistakes made by perceptron is at most 1/γ 2 Observations: 1. Lower the margin, more mistakes 2. May need more than one pass over the training data to get a classifier which makes no mistakes on the training set
10 What if data is not linearly separable? Ideally, we want to find a linear separator that makes the minimum number of mistakes on training data But this is NP Hard
11 Voted Perceptron Perceptron: 1. Initially: m = 1, w1 = y1x1 2. For t=2, 3,... If y t w m,x t 0 3. Output wm then: wm1 = wm yt xt m = m 1 Voted Perceptron: 1. Initially: m = 1, w1 = y1x1 2. For t=2, 3,... If Else: y t w m,x t 0 then: wm1 = wm yt xt m = m 1 cm = 1 cm = cm 1 3. Output (w1, c1), (w2, c2),..., (wm, cm)
12 Voted Perceptron Voted Perceptron: 1. Initially: m = 1, w1 = y1x1 2. For t=2, 3,... If Else: y t w m,x t 0 then: wm1 = wm yt xt m = m 1 cm = 1 cm = cm 1 3. Output (w1, c1), (w2, c2),..., (wm, cm) How to classify example x? Output: sign( m i=1 c i sign(w i,x)) Problem: Have to store all the classifiers
13 Averaged Perceptron Averaged Perceptron: 1. Initially: m = 1, w1 = y1x1 2. For t=2, 3,... If Else: y t w m,x t 0 then: wm1 = wm yt xt m = m 1 cm = 1 cm = cm 1 3. Output (w1, c1), (w2, c2),..., (wm, cm) How to classify example x? m Output: sign( c i w i,x) i=1
14 Voted vs. Averaged How to classify example x? Averaged: sign( Voted: sign( m i=1 m i=1 c i w i,x) c i sign(w i,x)) Example where voted and averaged are different: w1 = x1, w2 = x2, w3 = x3, c1 = c2 = c3 = 1 Averaged: x1 x2 x3 > 0 Voted: Any two of x1, x2, x3 > 0
15 Perceptron: An Example Data: ( (4, 0), 1), ( (1, 1), 1), ( (0, 1), 1), ( (2,2), 1) O w
16 Perceptron: An Example Data: ( (4, 0), 1), ( (1, 1), 1), ( (0, 1), 1), ( (2,2), 1) Step 1: w1 = (4, 0), c1 = 1 O w1 Decision Boundary
17 Perceptron: An Example Data: ( (4, 0), 1), ( (1, 1), 1), ( (0, 1), 1), ( (2,2), 1) Step 1: w1 = (4, 0), c1 = 1 Step 2: (x2, y2) = ( (1, 1), 1) x2 w1 O Decision Boundary
18 Perceptron: An Example Data: ( (4, 0), 1), ( (1, 1), 1), ( (0, 1), 1), ( (2,2), 1) Step 1: w1 = (4, 0), c1 = 1 Step 2: (x2, y2) = ( (1, 1), 1) y 2 w 1,x 2 < 0 Mistake! x2 w1 O Decision Boundary
19 Perceptron: An Example Data: ( (4, 0), 1), ( (1, 1), 1), ( (0, 1), 1), ( (2,2), 1) Step 1: w1 = (4, 0), c1 = 1 Step 2: (x2, y2) = ( (1, 1), 1) y 2 w 1,x 2 < 0 Mistake! Update: w2 = (3, 1), c2 = 1 x2 w1 O x2 w2 Decision Boundary
20 Perceptron: An Example Data: ( (4, 0), 1), ( (1, 1), 1), ( (0, 1), 1), ( (2,2), 1) Step 1: w1 = (4, 0), c1 = 1 Step 2: (x2, y2) = ( (1, 1), 1) y 2 w 1,x 2 < 0 Mistake! Update: w2 = (3, 1), c2 = 1 Step 3: (x3, y3) = ( (0, 1), 1) x3 x2 w1 O w2 Decision Boundary
21 Perceptron: An Example Data: ( (4, 0), 1), ( (1, 1), 1), ( (0, 1), 1), ( (2,2), 1) Step 1: w1 = (4, 0), c1 = 1 Step 2: (x2, y2) = ( (1, 1), 1) y 2 w 1,x 2 < 0 Mistake! Update: w2 = (3, 1), c2 = 1 Step 3: (x3, y3) = ( (0, 1), 1) y 3 w 2,x 3 > 0 Correct! x3 x2 O w2 Decision Boundary
22 Perceptron: An Example Data: ( (4, 0), 1), ( (1, 1), 1), ( (0, 1), 1), ( (2,2), 1) Step 1: w1 = (4, 0), c1 = 1 Step 2: (x2, y2) = ( (1, 1), 1) y 2 w 1,x 2 < 0 Mistake! Update: w2 = (3, 1), c2 = 1 Step 3: (x3, y3) = ( (0, 1), 1) y 3 w 2,x 3 > 0 Correct! Update: c2 = 2 x3 x2 O w2 Decision Boundary
23 Perceptron: An Example Data: ( (4, 0), 1), ( (1, 1), 1), ( (0, 1), 1), ( (2,2), 1) Step 1: w1 = (4, 0), c1 = 1 Step 2: (x2, y2) = ( (1, 1), 1) y 2 w 1,x 2 < 0 Mistake! Update: w2 = (3, 1), c2 = 1 Step 3: (x3, y3) = ( (0, 1), 1) y 3 w 2,x 3 > 0 Correct! Update: c2 = 2 Step 4: (x4, y4) = ( (2, 2), 1) x3 x2 O x4 w2 Decision Boundary
24 Perceptron: An Example Data: ( (4, 0), 1), ( (1, 1), 1), ( (0, 1), 1), ( (2,2), 1) Step 1: w1 = (4, 0), c1 = 1 Step 2: (x2, y2) = ( (1, 1), 1) y 2 w 1,x 2 < 0 Mistake! x3 x2 Update: w2 = (3, 1), c2 = 1 Step 3: (x3, y3) = ( (0, 1), 1) y 3 w 2,x 3 > 0 Correct! Update: c2 = 2 Step 4: (x4, y4) = ( (2, 2), 1) y 4 w 2,x 4 < 0 Mistake! O x4 w2 Decision Boundary
25 Perceptron: An Example Data: ( (4, 0), 1), ( (1, 1), 1), ( (0, 1), 1), ( (2,2), 1) Step 1: w1 = (4, 0), c1 = 1 Step 2: (x2, y2) = ( (1, 1), 1) y 2 w 1,x 2 < 0 Mistake! Update: w2 = (3, 1), c2 = 1 Step 3: (x3, y3) = ( (0, 1), 1) y 3 w 2,x 3 > 0 Correct! Decision Boundary Update: c2 = 2 Step 4: (x4, y4) = ( (2, 2), 1) y 4 w 2,x 4 < 0 Mistake! Update: w3 = (1, 3), c3 = 1 O x4 w2 w3
26 Perceptron: An Example Data: ( (4, 0), 1), ( (1, 1), 1), ( (0, 1), 1), ( (2,2), 1) Decision Boundary Step 1: w1 = (4, 0), c1 = 1 Step 2: (x2, y2) = ( (1, 1), 1) y 2 w 1,x 2 < 0 Mistake! Update: w2 = (3, 1), c2 = 1 Step 3: (x3, y3) = ( (0, 1), 1) y 3 w 2,x 3 > 0 Correct! Update: c2 = 2 Step 4: (x4, y4) = ( (2, 2), 1) y 4 w 2,x 4 < 0 Mistake! Update: w3 = (1, 3), c3 = 1 x4 O w2 Final Voted Perceptron: sign(sign(w 1,x)2sign(w 2,x)sign(w 3,x)) w3 Final Averaged Perceptron: sign(w 1 2w 2 w 3,x)
27 Summary Linear classification Perceptron Voted and Averaged Perceptron Some issues
28 Margins Max margin separator Separator Suppose data is linearly separable by a margin. Perceptron will stop once it finds a linear separator. But sometimes we would like to compute the maxmargin separator. This is done by Support Vector Machines (SVMs).
29 More General Linear Classification Given labeled data (xi, yi), i=1,..,n, where y is 1 or 1, Find a hyperplane to separate from We can transform general linear classification to linear classification by a hyperplane through the origin as follows: For each (xi, yi) (xi is ddimensional), construct: (zi, yi), where zi is the d1dimensional vector [1, xi] Any hyperplane through the origin in the zspace corresponds to a hyperplane (not necessarily through the origin) in xspace, and vice versa
30 Categorical Features Features with no inherent ordering Example: Gender = Male, Female State = Alaska, Hawaii, California Convert a categorical feature with k categories to a kdimensional vector: e.g, Alaska maps to [1, 0, 0], Hawaii to [0, 1, 0], California to [0, 0, 1]
31 Multiclass Linear Classification Main Idea: Reduce to multiple binary classification problems Onevsall: For k classes, solve k binary classification problems Class 1 vs. rest, Class 2 vs. rest,..., Class k vs. rest Allvsall: For k classes, solve k(k1)/2 binary classification problems Class i vs. j, for i=1,..,k, and j = i1,..,k To classify, combine by voting
32 The Confusion Matrix Mij = #examples that have label j but are classified as i Nj = #examples with label j Cij = Mij/Nj Indicates which classes are easy and hard to classify High diagonal entry: easy to classify High offdiagonal entry: high scope for confusion
33 Kernels
34 Feature Maps X2 Z2 φ X1 Z1 Z3 Data which is not linearly separable may be linearly separable in another feature space. In the figure, data is linearly separable in zspace: (Z1, Z2, Z3) = (X1 2, 2X1X2, X2 2 )
35 Feature Maps X2 Z2 φ X1 Z1 Z3 Let φ(x) be a feature map. For example: φ(x) = (x1 2, x1x2, x2 2 ) We want to run perceptron on the feature space φ(x)
36 Feature Maps X2 X1 What feature map will make this dataset linearly separable?
37 Feature Maps Let φ(x) be a feature map. For example: φ(x) = (x1 2, x1, x2) We want to run perceptron on the feature space φ(x) Many linear classification algorithms, including perceptron, SVMs, etc can be written in terms of dotproducts x, z We can simply change these dot products to φ(x), φ(z) For a feature map φ we define the corresponding kernel K: K(x,y) =φ(x), φ(y) Thus we can write the perceptron algorithm in terms of K
38 Kernels For a feature map φ we define the corresponding kernel K: K(x,y) =φ(x), φ(y) Examples: 1. K(x, z) =(x, z) 2
39 Kernels For a feature map φ we define the corresponding kernel K: K(x,y) =φ(x), φ(y) Examples: 1. K(x, z) =(x, z) 2 d K(x, z) = x i z i = d d x i x j z i z j = d (x i x j )(z i z j ) i=1 x i z i d i=1 i=1 j=1 i,j=1 For d=3, φ(x) = [ x1 2, x1x2, x1x3, x2x1, x2 2, x2x3, x3x1, x3x2, x3 2 ] T Time to compute K directly = O(d) Time to compute K though feature map = O(d 2 )
40 Kernels For a feature map φ we define the corresponding kernel K: K(x,y) =φ(x), φ(y) Examples: K(x, z) =(x, z) 2 K(x, z) =(x, z c) 2 In Class Problem: What is the feature map for this kernel?
Outline. Basic concepts: SVM and kernels SVM primal/dual problems. Chih-Jen Lin (National Taiwan Univ.) 1 / 22
Outline Basic concepts: SVM and kernels SVM primal/dual problems Chih-Jen Lin (National Taiwan Univ.) 1 / 22 Outline Basic concepts: SVM and kernels Basic concepts: SVM and kernels SVM primal/dual problems
More informationLinear smoother. ŷ = S y. where s ij = s ij (x) e.g. s ij = diag(l i (x))
Linear smoother ŷ = S y where s ij = s ij (x) e.g. s ij = diag(l i (x)) 2 Online Learning: LMS and Perceptrons Partially adapted from slides by Ryan Gabbard and Mitch Marcus (and lots original slides by
More informationMore about the Perceptron
More about the Perceptron CMSC 422 MARINE CARPUAT marine@cs.umd.edu Credit: figures by Piyush Rai and Hal Daume III Recap: Perceptron for binary classification Classifier = hyperplane that separates positive
More informationSupport Vector Machines. Introduction to Data Mining, 2 nd Edition by Tan, Steinbach, Karpatne, Kumar
Data Mining Support Vector Machines Introduction to Data Mining, 2 nd Edition by Tan, Steinbach, Karpatne, Kumar 02/03/2018 Introduction to Data Mining 1 Support Vector Machines Find a linear hyperplane
More informationData Mining. Linear & nonlinear classifiers. Hamid Beigy. Sharif University of Technology. Fall 1396
Data Mining Linear & nonlinear classifiers Hamid Beigy Sharif University of Technology Fall 1396 Hamid Beigy (Sharif University of Technology) Data Mining Fall 1396 1 / 31 Table of contents 1 Introduction
More informationCS4495/6495 Introduction to Computer Vision. 8C-L3 Support Vector Machines
CS4495/6495 Introduction to Computer Vision 8C-L3 Support Vector Machines Discriminative classifiers Discriminative classifiers find a division (surface) in feature space that separates the classes Several
More informationLinear & nonlinear classifiers
Linear & nonlinear classifiers Machine Learning Hamid Beigy Sharif University of Technology Fall 1394 Hamid Beigy (Sharif University of Technology) Linear & nonlinear classifiers Fall 1394 1 / 34 Table
More informationSupport Vector Machine. Industrial AI Lab.
Support Vector Machine Industrial AI Lab. Classification (Linear) Autonomously figure out which category (or class) an unknown item should be categorized into Number of categories / classes Binary: 2 different
More informationLinear & nonlinear classifiers
Linear & nonlinear classifiers Machine Learning Hamid Beigy Sharif University of Technology Fall 1396 Hamid Beigy (Sharif University of Technology) Linear & nonlinear classifiers Fall 1396 1 / 44 Table
More informationCOMS 4721: Machine Learning for Data Science Lecture 10, 2/21/2017
COMS 4721: Machine Learning for Data Science Lecture 10, 2/21/2017 Prof. John Paisley Department of Electrical Engineering & Data Science Institute Columbia University FEATURE EXPANSIONS FEATURE EXPANSIONS
More informationSupport Vector Machine (SVM) & Kernel CE-717: Machine Learning Sharif University of Technology. M. Soleymani Fall 2012
Support Vector Machine (SVM) & Kernel CE-717: Machine Learning Sharif University of Technology M. Soleymani Fall 2012 Linear classifier Which classifier? x 2 x 1 2 Linear classifier Margin concept x 2
More informationEE 511 Online Learning Perceptron
Slides adapted from Ali Farhadi, Mari Ostendorf, Pedro Domingos, Carlos Guestrin, and Luke Zettelmoyer, Kevin Jamison EE 511 Online Learning Perceptron Instructor: Hanna Hajishirzi hannaneh@washington.edu
More informationSupport Vector Machine. Industrial AI Lab. Prof. Seungchul Lee
Support Vector Machine Industrial AI Lab. Prof. Seungchul Lee Classification (Linear) Autonomously figure out which category (or class) an unknown item should be categorized into Number of categories /
More informationJeff Howbert Introduction to Machine Learning Winter
Classification / Regression Support Vector Machines Jeff Howbert Introduction to Machine Learning Winter 2012 1 Topics SVM classifiers for linearly separable classes SVM classifiers for non-linearly separable
More informationSupport Vector Machine
Support Vector Machine Kernel: Kernel is defined as a function returning the inner product between the images of the two arguments k(x 1, x 2 ) = ϕ(x 1 ), ϕ(x 2 ) k(x 1, x 2 ) = k(x 2, x 1 ) modularity-
More informationCOMS 4771 Introduction to Machine Learning. Nakul Verma
COMS 4771 Introduction to Machine Learning Nakul Verma Announcements HW1 due next lecture Project details are available decide on the group and topic by Thursday Last time Generative vs. Discriminative
More informationMIRA, SVM, k-nn. Lirong Xia
MIRA, SVM, k-nn Lirong Xia Linear Classifiers (perceptrons) Inputs are feature values Each feature has a weight Sum is the activation activation w If the activation is: Positive: output +1 Negative, output
More informationCS798: Selected topics in Machine Learning
CS798: Selected topics in Machine Learning Support Vector Machine Jakramate Bootkrajang Department of Computer Science Chiang Mai University Jakramate Bootkrajang CS798: Selected topics in Machine Learning
More informationSupport vector machines Lecture 4
Support vector machines Lecture 4 David Sontag New York University Slides adapted from Luke Zettlemoyer, Vibhav Gogate, and Carlos Guestrin Q: What does the Perceptron mistake bound tell us? Theorem: The
More informationThe Perceptron Algorithm
The Perceptron Algorithm Greg Grudic Greg Grudic Machine Learning Questions? Greg Grudic Machine Learning 2 Binary Classification A binary classifier is a mapping from a set of d inputs to a single output
More informationThe Perceptron Algorithm
The Perceptron Algorithm Machine Learning Spring 2018 The slides are mainly from Vivek Srikumar 1 Outline The Perceptron Algorithm Perceptron Mistake Bound Variants of Perceptron 2 Where are we? The Perceptron
More informationPattern Recognition 2018 Support Vector Machines
Pattern Recognition 2018 Support Vector Machines Ad Feelders Universiteit Utrecht Ad Feelders ( Universiteit Utrecht ) Pattern Recognition 1 / 48 Support Vector Machines Ad Feelders ( Universiteit Utrecht
More information10/05/2016. Computational Methods for Data Analysis. Massimo Poesio SUPPORT VECTOR MACHINES. Support Vector Machines Linear classifiers
Computational Methods for Data Analysis Massimo Poesio SUPPORT VECTOR MACHINES Support Vector Machines Linear classifiers 1 Linear Classifiers denotes +1 denotes -1 w x + b>0 f(x,w,b) = sign(w x + b) How
More informationLearning with kernels and SVM
Learning with kernels and SVM Šámalova chata, 23. května, 2006 Petra Kudová Outline Introduction Binary classification Learning with Kernels Support Vector Machines Demo Conclusion Learning from data find
More informationFinal Exam. 1 True or False (15 Points)
10-606 Final Exam Submit by Oct. 16, 2017 11:59pm EST Please submit early, and update your submission if you want to make changes. Do not wait to the last minute to submit: we reserve the right not to
More information6.036 midterm review. Wednesday, March 18, 15
6.036 midterm review 1 Topics covered supervised learning labels available unsupervised learning no labels available semi-supervised learning some labels available - what algorithms have you learned that
More informationLinear Classification and SVM. Dr. Xin Zhang
Linear Classification and SVM Dr. Xin Zhang Email: eexinzhang@scut.edu.cn What is linear classification? Classification is intrinsically non-linear It puts non-identical things in the same class, so a
More informationLINEAR CLASSIFICATION, PERCEPTRON, LOGISTIC REGRESSION, SVC, NAÏVE BAYES. Supervised Learning
LINEAR CLASSIFICATION, PERCEPTRON, LOGISTIC REGRESSION, SVC, NAÏVE BAYES Supervised Learning Linear vs non linear classifiers In K-NN we saw an example of a non-linear classifier: the decision boundary
More informationDEPARTMENT OF COMPUTER SCIENCE Autumn Semester MACHINE LEARNING AND ADAPTIVE INTELLIGENCE
Data Provided: None DEPARTMENT OF COMPUTER SCIENCE Autumn Semester 203 204 MACHINE LEARNING AND ADAPTIVE INTELLIGENCE 2 hours Answer THREE of the four questions. All questions carry equal weight. Figures
More informationMachine Learning Lecture 6 Note
Machine Learning Lecture 6 Note Compiled by Abhi Ashutosh, Daniel Chen, and Yijun Xiao February 16, 2016 1 Pegasos Algorithm The Pegasos Algorithm looks very similar to the Perceptron Algorithm. In fact,
More informationMachine Learning. Kernels. Fall (Kernels, Kernelized Perceptron and SVM) Professor Liang Huang. (Chap. 12 of CIML)
Machine Learning Fall 2017 Kernels (Kernels, Kernelized Perceptron and SVM) Professor Liang Huang (Chap. 12 of CIML) Nonlinear Features x4: -1 x1: +1 x3: +1 x2: -1 Concatenated (combined) features XOR:
More informationKernelized Perceptron Support Vector Machines
Kernelized Perceptron Support Vector Machines Emily Fox University of Washington February 13, 2017 What is the perceptron optimizing? 1 The perceptron algorithm [Rosenblatt 58, 62] Classification setting:
More informationLogistic regression and linear classifiers COMS 4771
Logistic regression and linear classifiers COMS 4771 1. Prediction functions (again) Learning prediction functions IID model for supervised learning: (X 1, Y 1),..., (X n, Y n), (X, Y ) are iid random
More informationCIS 520: Machine Learning Oct 09, Kernel Methods
CIS 520: Machine Learning Oct 09, 207 Kernel Methods Lecturer: Shivani Agarwal Disclaimer: These notes are designed to be a supplement to the lecture They may or may not cover all the material discussed
More informationLinear vs Non-linear classifier. CS789: Machine Learning and Neural Network. Introduction
Linear vs Non-linear classifier CS789: Machine Learning and Neural Network Support Vector Machine Jakramate Bootkrajang Department of Computer Science Chiang Mai University Linear classifier is in the
More informationFrom Binary to Multiclass Classification. CS 6961: Structured Prediction Spring 2018
From Binary to Multiclass Classification CS 6961: Structured Prediction Spring 2018 1 So far: Binary Classification We have seen linear models Learning algorithms Perceptron SVM Logistic Regression Prediction
More informationNon-Bayesian Classifiers Part II: Linear Discriminants and Support Vector Machines
Non-Bayesian Classifiers Part II: Linear Discriminants and Support Vector Machines Selim Aksoy Department of Computer Engineering Bilkent University saksoy@cs.bilkent.edu.tr CS 551, Fall 2018 CS 551, Fall
More informationMulticlass Classification-1
CS 446 Machine Learning Fall 2016 Oct 27, 2016 Multiclass Classification Professor: Dan Roth Scribe: C. Cheng Overview Binary to multiclass Multiclass SVM Constraint classification 1 Introduction Multiclass
More informationSVMs: nonlinearity through kernels
Non-separable data e-8. Support Vector Machines 8.. The Optimal Hyperplane Consider the following two datasets: SVMs: nonlinearity through kernels ER Chapter 3.4, e-8 (a) Few noisy data. (b) Nonlinearly
More informationSupport Vector Machines
Support Vector Machines Hypothesis Space variable size deterministic continuous parameters Learning Algorithm linear and quadratic programming eager batch SVMs combine three important ideas Apply optimization
More informationEvaluation. Andrea Passerini Machine Learning. Evaluation
Andrea Passerini passerini@disi.unitn.it Machine Learning Basic concepts requires to define performance measures to be optimized Performance of learning algorithms cannot be evaluated on entire domain
More informationSupport Vector Machines II. CAP 5610: Machine Learning Instructor: Guo-Jun QI
Support Vector Machines II CAP 5610: Machine Learning Instructor: Guo-Jun QI 1 Outline Linear SVM hard margin Linear SVM soft margin Non-linear SVM Application Linear Support Vector Machine An optimization
More information1 Machine Learning Concepts (16 points)
CSCI 567 Fall 2018 Midterm Exam DO NOT OPEN EXAM UNTIL INSTRUCTED TO DO SO PLEASE TURN OFF ALL CELL PHONES Problem 1 2 3 4 5 6 Total Max 16 10 16 42 24 12 120 Points Please read the following instructions
More informationSupport Vector and Kernel Methods
SIGIR 2003 Tutorial Support Vector and Kernel Methods Thorsten Joachims Cornell University Computer Science Department tj@cs.cornell.edu http://www.joachims.org 0 Linear Classifiers Rules of the Form:
More informationSupport Vector Machine & Its Applications
Support Vector Machine & Its Applications A portion (1/3) of the slides are taken from Prof. Andrew Moore s SVM tutorial at http://www.cs.cmu.edu/~awm/tutorials Mingyue Tan The University of British Columbia
More informationBrief Introduction to Machine Learning
Brief Introduction to Machine Learning Yuh-Jye Lee Lab of Data Science and Machine Intelligence Dept. of Applied Math. at NCTU August 29, 2016 1 / 49 1 Introduction 2 Binary Classification 3 Support Vector
More informationReview: Support vector machines. Machine learning techniques and image analysis
Review: Support vector machines Review: Support vector machines Margin optimization min (w,w 0 ) 1 2 w 2 subject to y i (w 0 + w T x i ) 1 0, i = 1,..., n. Review: Support vector machines Margin optimization
More informationEvaluation requires to define performance measures to be optimized
Evaluation Basic concepts Evaluation requires to define performance measures to be optimized Performance of learning algorithms cannot be evaluated on entire domain (generalization error) approximation
More informationApplied Machine Learning Annalisa Marsico
Applied Machine Learning Annalisa Marsico OWL RNA Bionformatics group Max Planck Institute for Molecular Genetics Free University of Berlin 29 April, SoSe 2015 Support Vector Machines (SVMs) 1. One of
More informationCS 188: Artificial Intelligence. Outline
CS 188: Artificial Intelligence Lecture 21: Perceptrons Pieter Abbeel UC Berkeley Many slides adapted from Dan Klein. Outline Generative vs. Discriminative Binary Linear Classifiers Perceptron Multi-class
More informationCSC321 Lecture 4 The Perceptron Algorithm
CSC321 Lecture 4 The Perceptron Algorithm Roger Grosse and Nitish Srivastava January 17, 2017 Roger Grosse and Nitish Srivastava CSC321 Lecture 4 The Perceptron Algorithm January 17, 2017 1 / 1 Recap:
More informationNeural networks and support vector machines
Neural netorks and support vector machines Perceptron Input x 1 Weights 1 x 2 x 3... x D 2 3 D Output: sgn( x + b) Can incorporate bias as component of the eight vector by alays including a feature ith
More informationSVMs: Non-Separable Data, Convex Surrogate Loss, Multi-Class Classification, Kernels
SVMs: Non-Separable Data, Convex Surrogate Loss, Multi-Class Classification, Kernels Karl Stratos June 21, 2018 1 / 33 Tangent: Some Loose Ends in Logistic Regression Polynomial feature expansion in logistic
More informationKernels. Machine Learning CSE446 Carlos Guestrin University of Washington. October 28, Carlos Guestrin
Kernels Machine Learning CSE446 Carlos Guestrin University of Washington October 28, 2013 Carlos Guestrin 2005-2013 1 Linear Separability: More formally, Using Margin Data linearly separable, if there
More informationLinear Classifiers. Michael Collins. January 18, 2012
Linear Classifiers Michael Collins January 18, 2012 Today s Lecture Binary classification problems Linear classifiers The perceptron algorithm Classification Problems: An Example Goal: build a system that
More informationDownload the following data sets from the UC Irvine ML repository:
Mehryar Mohri Introduction to Machine Learning Courant Institute of Mathematical Sciences Homework assignment 2 Solution (written by Andres Muñoz) Perceptron Algorithm Download the following data sets
More informationMulti-class SVMs. Lecture 17: Aykut Erdem April 2016 Hacettepe University
Multi-class SVMs Lecture 17: Aykut Erdem April 2016 Hacettepe University Administrative We will have a make-up lecture on Saturday April 23, 2016. Project progress reports are due April 21, 2016 2 days
More informationLecture 7: Kernels for Classification and Regression
Lecture 7: Kernels for Classification and Regression CS 194-10, Fall 2011 Laurent El Ghaoui EECS Department UC Berkeley September 15, 2011 Outline Outline A linear regression problem Linear auto-regressive
More informationSupport Vector Machines and Kernel Methods
2018 CS420 Machine Learning, Lecture 3 Hangout from Prof. Andrew Ng. http://cs229.stanford.edu/notes/cs229-notes3.pdf Support Vector Machines and Kernel Methods Weinan Zhang Shanghai Jiao Tong University
More informationSupport Vector Machine (SVM) and Kernel Methods
Support Vector Machine (SVM) and Kernel Methods CE-717: Machine Learning Sharif University of Technology Fall 2014 Soleymani Outline Margin concept Hard-Margin SVM Soft-Margin SVM Dual Problems of Hard-Margin
More informationCS145: INTRODUCTION TO DATA MINING
CS145: INTRODUCTION TO DATA MINING 5: Vector Data: Support Vector Machine Instructor: Yizhou Sun yzsun@cs.ucla.edu October 18, 2017 Homework 1 Announcements Due end of the day of this Thursday (11:59pm)
More informationSupport Vector Machines. CSE 6363 Machine Learning Vassilis Athitsos Computer Science and Engineering Department University of Texas at Arlington
Support Vector Machines CSE 6363 Machine Learning Vassilis Athitsos Computer Science and Engineering Department University of Texas at Arlington 1 A Linearly Separable Problem Consider the binary classification
More informationPerceptron Revisited: Linear Separators. Support Vector Machines
Support Vector Machines Perceptron Revisited: Linear Separators Binary classification can be viewed as the task of separating classes in feature space: w T x + b > 0 w T x + b = 0 w T x + b < 0 Department
More informationIntroduction to Support Vector Machines
Introduction to Support Vector Machines Hsuan-Tien Lin Learning Systems Group, California Institute of Technology Talk in NTU EE/CS Speech Lab, November 16, 2005 H.-T. Lin (Learning Systems Group) Introduction
More informationMachine Learning Linear Models
Machine Learning Linear Models Outline II - Linear Models 1. Linear Regression (a) Linear regression: History (b) Linear regression with Least Squares (c) Matrix representation and Normal Equation Method
More informationDeep Learning for Computer Vision
Deep Learning for Computer Vision Lecture 4: Curse of Dimensionality, High Dimensional Feature Spaces, Linear Classifiers, Linear Regression, Python, and Jupyter Notebooks Peter Belhumeur Computer Science
More informationMachine Learning. Lecture 3: Logistic Regression. Feng Li.
Machine Learning Lecture 3: Logistic Regression Feng Li fli@sdu.edu.cn https://funglee.github.io School of Computer Science and Technology Shandong University Fall 2016 Logistic Regression Classification
More informationSimple Neural Nets For Pattern Classification
CHAPTER 2 Simple Neural Nets For Pattern Classification Neural Networks General Discussion One of the simplest tasks that neural nets can be trained to perform is pattern classification. In pattern classification
More informationKernel Methods and Support Vector Machines
Kernel Methods and Support Vector Machines Oliver Schulte - CMPT 726 Bishop PRML Ch. 6 Support Vector Machines Defining Characteristics Like logistic regression, good for continuous input features, discrete
More informationCSE 151 Machine Learning. Instructor: Kamalika Chaudhuri
CSE 151 Machine Learning Instructor: Kamalika Chaudhuri Ensemble Learning How to combine multiple classifiers into a single one Works well if the classifiers are complementary This class: two types of
More informationSupport Vector Machine (SVM) and Kernel Methods
Support Vector Machine (SVM) and Kernel Methods CE-717: Machine Learning Sharif University of Technology Fall 2015 Soleymani Outline Margin concept Hard-Margin SVM Soft-Margin SVM Dual Problems of Hard-Margin
More informationCSE 546 Final Exam, Autumn 2013
CSE 546 Final Exam, Autumn 0. Personal info: Name: Student ID: E-mail address:. There should be 5 numbered pages in this exam (including this cover sheet).. You can use any material you brought: any book,
More informationLecture 9: Large Margin Classifiers. Linear Support Vector Machines
Lecture 9: Large Margin Classifiers. Linear Support Vector Machines Perceptrons Definition Perceptron learning rule Convergence Margin & max margin classifiers (Linear) support vector machines Formulation
More informationSecurity Analytics. Topic 6: Perceptron and Support Vector Machine
Security Analytics Topic 6: Perceptron and Support Vector Machine Purdue University Prof. Ninghui Li Based on slides by Prof. Jenifer Neville and Chris Clifton Readings Principle of Data Mining Chapter
More informationSupport Vector Machines
Two SVM tutorials linked in class website (please, read both): High-level presentation with applications (Hearst 1998) Detailed tutorial (Burges 1998) Support Vector Machines Machine Learning 10701/15781
More informationFoundation of Intelligent Systems, Part I. SVM s & Kernel Methods
Foundation of Intelligent Systems, Part I SVM s & Kernel Methods mcuturi@i.kyoto-u.ac.jp FIS - 2013 1 Support Vector Machines The linearly-separable case FIS - 2013 2 A criterion to select a linear classifier:
More informationMachine Learning. Linear Models. Fabio Vandin October 10, 2017
Machine Learning Linear Models Fabio Vandin October 10, 2017 1 Linear Predictors and Affine Functions Consider X = R d Affine functions: L d = {h w,b : w R d, b R} where ( d ) h w,b (x) = w, x + b = w
More informationLinear discriminant functions
Andrea Passerini passerini@disi.unitn.it Machine Learning Discriminative learning Discriminative vs generative Generative learning assumes knowledge of the distribution governing the data Discriminative
More informationSparse Kernel Machines - SVM
Sparse Kernel Machines - SVM Henrik I. Christensen Robotics & Intelligent Machines @ GT Georgia Institute of Technology, Atlanta, GA 30332-0280 hic@cc.gatech.edu Henrik I. Christensen (RIM@GT) Support
More informationInstance-based Learning CE-717: Machine Learning Sharif University of Technology. M. Soleymani Fall 2016
Instance-based Learning CE-717: Machine Learning Sharif University of Technology M. Soleymani Fall 2016 Outline Non-parametric approach Unsupervised: Non-parametric density estimation Parzen Windows Kn-Nearest
More informationMachine learning comes from Bayesian decision theory in statistics. There we want to minimize the expected value of the loss function.
Bayesian learning: Machine learning comes from Bayesian decision theory in statistics. There we want to minimize the expected value of the loss function. Let y be the true label and y be the predicted
More informationSupport Vector Machines (SVM) in bioinformatics. Day 1: Introduction to SVM
1 Support Vector Machines (SVM) in bioinformatics Day 1: Introduction to SVM Jean-Philippe Vert Bioinformatics Center, Kyoto University, Japan Jean-Philippe.Vert@mines.org Human Genome Center, University
More informationSupport Vector Machines
EE 17/7AT: Optimization Models in Engineering Section 11/1 - April 014 Support Vector Machines Lecturer: Arturo Fernandez Scribe: Arturo Fernandez 1 Support Vector Machines Revisited 1.1 Strictly) Separable
More informationMachine Learning 4771
Machine Learning 477 Instructor: Tony Jebara Topic 5 Generalization Guarantees VC-Dimension Nearest Neighbor Classification (infinite VC dimension) Structural Risk Minimization Support Vector Machines
More informationMistake Bound Model, Halving Algorithm, Linear Classifiers, & Perceptron
Stat 928: Statistical Learning Theory Lecture: 18 Mistake Bound Model, Halving Algorithm, Linear Classifiers, & Perceptron Instructor: Sham Kakade 1 Introduction This course will be divided into 2 parts.
More informationLinear Support Vector Machine. Classification. Linear SVM. Huiping Cao. Huiping Cao, Slide 1/26
Huiping Cao, Slide 1/26 Classification Linear SVM Huiping Cao linear hyperplane (decision boundary) that will separate the data Huiping Cao, Slide 2/26 Support Vector Machines rt Vector Find a linear Machines
More informationWarm up: risk prediction with logistic regression
Warm up: risk prediction with logistic regression Boss gives you a bunch of data on loans defaulting or not: {(x i,y i )} n i= x i 2 R d, y i 2 {, } You model the data as: P (Y = y x, w) = + exp( yw T
More informationSupport Vector Machines.
Support Vector Machines www.cs.wisc.edu/~dpage 1 Goals for the lecture you should understand the following concepts the margin slack variables the linear support vector machine nonlinear SVMs the kernel
More informationDeviations from linear separability. Kernel methods. Basis expansion for quadratic boundaries. Adding new features Systematic deviation
Deviations from linear separability Kernel methods CSE 250B Noise Find a separator that minimizes a convex loss function related to the number of mistakes. e.g. SVM, logistic regression. Systematic deviation
More informationIntroduction to Machine Learning
1, DATA11002 Introduction to Machine Learning Lecturer: Teemu Roos TAs: Ville Hyvönen and Janne Leppä-aho Department of Computer Science University of Helsinki (based in part on material by Patrik Hoyer
More informationLogistic Regression Logistic
Case Study 1: Estimating Click Probabilities L2 Regularization for Logistic Regression Machine Learning/Statistics for Big Data CSE599C1/STAT592, University of Washington Carlos Guestrin January 10 th,
More informationIntroduction to Support Vector Machines
Introduction to Support Vector Machines Shivani Agarwal Support Vector Machines (SVMs) Algorithm for learning linear classifiers Motivated by idea of maximizing margin Efficient extension to non-linear
More informationMachine Learning. Linear Models. Fabio Vandin October 10, 2017
Machine Learning Linear Models Fabio Vandin October 10, 2017 1 Linear Predictors and Affine Functions Consider X = R d Affine functions: L d = {h w,b : w R d, b R} where ( d ) h w,b (x) = w, x + b = w
More informationPerceptron. Subhransu Maji. CMPSCI 689: Machine Learning. 3 February February 2015
Perceptron Subhransu Maji CMPSCI 689: Machine Learning 3 February 2015 5 February 2015 So far in the class Decision trees Inductive bias: use a combination of small number of features Nearest neighbor
More informationMidterm: CS 6375 Spring 2015 Solutions
Midterm: CS 6375 Spring 2015 Solutions The exam is closed book. You are allowed a one-page cheat sheet. Answer the questions in the spaces provided on the question sheets. If you run out of room for an
More informationMachine Learning A Geometric Approach
Machine Learning A Geometric Approach Linear Classification: Perceptron Professor Liang Huang some slides from Alex Smola (CMU) Perceptron Frank Rosenblatt deep learning multilayer perceptron linear regression
More informationMachine Learning. Support Vector Machines. Manfred Huber
Machine Learning Support Vector Machines Manfred Huber 2015 1 Support Vector Machines Both logistic regression and linear discriminant analysis learn a linear discriminant function to separate the data
More information6.867 Machine learning
6.867 Machine learning Mid-term eam October 8, 6 ( points) Your name and MIT ID: .5.5 y.5 y.5 a).5.5 b).5.5.5.5 y.5 y.5 c).5.5 d).5.5 Figure : Plots of linear regression results with different types of
More informationIntroduction to Machine Learning
Introduction to Machine Learning Kernel Methods Varun Chandola Computer Science & Engineering State University of New York at Buffalo Buffalo, NY, USA chandola@buffalo.edu Chandola@UB CSE 474/574 1 / 21
More informationMachine Learning Practice Page 2 of 2 10/28/13
Machine Learning 10-701 Practice Page 2 of 2 10/28/13 1. True or False Please give an explanation for your answer, this is worth 1 pt/question. (a) (2 points) No classifier can do better than a naive Bayes
More information