Introduction to Machine Learning

Introduction to Machine Learning Machine Learning: Jordan Boyd-Graber University of Maryland SUPPORT VECTOR MACHINES Slides adapted from Tom Mitchell, Eric Xing, and Lauren Hannah Machine Learning: Jordan Boyd-Graber UMD Introduction to Machine Learning 1 / 29

Roadmap Classification: machines labeling data for us Previously: naïve Bayes and logistic regression This time: SVMs (another) example of linear classifier State-of-the-art classification Good theoretical properties Machine Learning: Jordan Boyd-Graber UMD Introduction to Machine Learning 2 / 29

Thinking Geometrically Suppose you have two classes: vacations and sports Suppose you have four documents Sports Doc 1 : {ball, ball, ball, travel} Doc 2 : {ball, ball} What does this look like in vector space? Vacations Doc 3 : {travel, ball, travel} Doc 4 : {travel} Machine Learning: Jordan Boyd-Graber UMD Introduction to Machine Learning 3 / 29

Put the documents in vector space Travel 3 2 1 0 1 2 3 Ball Machine Learning: Jordan Boyd-Graber UMD Introduction to Machine Learning 4 / 29

Vector space representation of documents Each document is a vector, one component for each term. Terms are axes. High dimensionality: 10,000s of dimensions and more How can we do classification in this space? Machine Learning: Jordan Boyd-Graber UMD Introduction to Machine Learning 5 / 29

Vector space classification As before, the training set is a set of documents, each labeled with its class. In vector space classification, this set corresponds to a labeled set of points or vectors in the vector space. Premise 1: Documents in the same class form a contiguous region. Premise 2: Documents from different classes don t overlap. We define lines, surfaces, hypersurfaces to divide regions. Machine Learning: Jordan Boyd-Graber UMD Introduction to Machine Learning 6 / 29

Recap Vector space classification Linear classifiers Support Vector Machines D Classes in the vector space Classes in the vector space UK China Kenya x x x x Should the document be assigned to China, UK or Machine Learning: Jordan Boyd-Graber UMD Introduction to Machine Learning 7 / 29

Recap Vector space classification Linear classifiers Support Vector Machines D Classes in the vector space Classes in the vector space UK China Kenya Should the document Should be the assigned document to China, be assigned UK or Kenya? to China, UK or x x x x Machine Learning: Jordan Boyd-Graber UMD Introduction to Machine Learning 7 / 29

Recap Vector space classification Linear classifiers Support Vector Machines Dis Classes in the vector space Classes in the vector space UK China Kenya Find separators between the classes Find separators between the classes x x x x Machine Learning: Jordan Boyd-Graber UMD Introduction to Machine Learning 7 / 29

Recap Vector space classification Linear classifiers Support Vector Machines Discu Classes in the vector space Classes in the vector space UK China Kenya Find separators between the classes Find separators between the classes x x x x Machine Learning: Jordan Boyd-Graber UMD Introduction to Machine Learning 7 / 29

Recap Vector space classification Linear classifiers Support Vector Machines Discus Classes in the vector space Classes in the vector space UK China Kenya Find separators between the classes Based on these separators: should be assigned to China x x x x Machine Learning: Jordan Boyd-Graber UMD Introduction to Machine Learning 7 / 29

Recap Vector space classification Linear classifiers Support Vector Machines Discus Classes in the vector space Classes in the vector space UK China Kenya Find separators between the classes How do we find separators that do a good job at classifying new documents like? Main topic of today x x x x Machine Learning: Jordan Boyd-Graber UMD Introduction to Machine Learning 7 / 29

Linear Classifiers Linear classifiers Definition: A linear classifier computes a linear combination or weighted sum β i ix i of the feature values. Classification decision: β i ix i > β 0? (β 0 is our bias)... where β 0 (the threshold) is a parameter. We call this the separator or decision boundary. We find the separator based on training set. Methods for finding separator: logistic regression, naïve Bayes, linear SVM Assumption: The classes are linearly separable. Machine Learning: Jordan Boyd-Graber UMD Introduction to Machine Learning 8 / 29

Linear Classifiers Linear classifiers Definition: A linear classifier computes a linear combination or weighted sum β i ix i of the feature values. Classification decision: β i ix i > β 0? (β 0 is our bias)... where β 0 (the threshold) is a parameter. We call this the separator or decision boundary. We find the separator based on training set. Methods for finding separator: logistic regression, naïve Bayes, linear SVM Assumption: The classes are linearly separable. Before, we just talked about equations. What s the geometric intuition? Machine Learning: Jordan Boyd-Graber UMD Introduction to Machine Learning 8 / 29

Linear Classifiers Alinearclassifierin1D A linear classifier in 1D Alinearclassifierin1Dis apointx described by the equation w 1 d 1 = θ x = θ/w 1 A linear classifier in 1D is a point x described by the equation β 1 x 1 = β 0 Schütze: Support vector machines 15 / 55 Machine Learning: Jordan Boyd-Graber UMD Introduction to Machine Learning 9 / 29

Linear Classifiers Alinearclassifierin1D A linear classifier in 1D Alinearclassifierin1Dis apointx described by the equation w 1 d 1 = θ x = θ/w 1 A linear classifier in 1D is a point x described by the equation β 1 x 1 = β 0 x = β0 /β 1 Schütze: Support vector machines 15 / 55 Machine Learning: Jordan Boyd-Graber UMD Introduction to Machine Learning 9 / 29

Linear Classifiers Alinearclassifierin1D A linear classifier in 1D Alinearclassifierin1Dis apointx described by the equation w 1 d 1 = θ x = θ/w 1 Points (d 1 )withw 1 d 1 θ are in the A linear class c. classifier in 1D is a point x described by the equation β 1 x 1 = β 0 x = β0 /β 1 Points (x1 ) with β 1 x 1 β 0 are in the class c. Schütze: Support vector machines 15 / 55 Machine Learning: Jordan Boyd-Graber UMD Introduction to Machine Learning 9 / 29

Linear Classifiers Alinearclassifierin1D A linear classifier in 1D Alinearclassifierin1Dis apointx described by the equation w 1 d 1 = θ x = θ/w 1 Points A (d linear 1 )withw classifier 1 d 1 θ in 1D is a are inpoint the class x described c. by the Points (d 1 )withw 1 d 1 < θ equation β are in the complement 1 x 1 = β 0 class c. x = β0 /β 1 Schütze: Support vector machines 15 / 55 Points (x1 ) with β 1 x 1 β 0 are in the class c. Points (x1 ) with β 1 x 1 < β 0 are in the complement class c. Machine Learning: Jordan Boyd-Graber UMD Introduction to Machine Learning 9 / 29

Linear Classifiers Recap Vector space classification Linear classifiers Support Vector Machines Discussion AAlinearclassifierin2D Alinearclassifierin2Dis A classifier in 2D is a alinedescribedbythe line described by the equation equation w 1 d 1 + w 2 d 2 = θ β 1 x 1 + β 2 x 2 = β 0 Example for a 2D linear classifier Example for a 2D linear Points (d 1 d 2 )with classifier w 1 d 1 + w 2 d 2 θ are in the class Points c. (x1 x 2 ) with β 1 x 1 + β 2 x 2 β 0 are in the class c. Schütze: Support vector machines 16 / 55 Machine Learning: Jordan Boyd-Graber UMD Introduction to Machine Learning 10 / 29

Linear Classifiers Recap Vector space classification Linear classifiers Support Vector Machines Discussion Alinearclassifierin2D A linear classifier in 2D Alinearclassifierin2Dis alinedescribedbythe A linear classifier in 2D is a equationline w 1 ddescribed 1 + w 2 d 2 = θby the equation Exampleβ for a 2D linear 1 x 1 + β 2 x 2 = β 0 classifier Points (d Example 1 d 2 )with for a 2D linear w 1 d 1 + classifier w 2 d 2 θ are in the class c. Points (x1 x 2 ) with Points (d 1 d 2 )with w 1 d 1 + βw 21 dx 21 < + θβare 2 x 2 in β 0 are in the the complement class c. class c. Schütze: Support vector machines 16 / 55 Points (x1 x 2 ) with β 1 x 1 + β 2 x 2 < β 0 are in the complement class c. Machine Learning: Jordan Boyd-Graber UMD Introduction to Machine Learning 10 / 29

Linear Classifiers Recap Vector space classification Linear classifiers Support Vector Machines Discussion A Alinearclassifierin3D Alinearclassifierin3Dis aplanedescribedbythe equation w 1 d 1 + w 2 d 2 + w 3 d 3 = θ A linear classifier in 3D is a plane described by the equation β 1 x 1 + β 2 x 2 + β 3 x 3 = β 0 Schütze: Support vector machines 17 / 55 Machine Learning: Jordan Boyd-Graber UMD Introduction to Machine Learning 11 / 29

Linear Classifiers Recap Vector space classification Linear classifiers Support Vector Machines Discussion A Alinearclassifierin3D Alinearclassifierin3Dis aplanedescribedbythe equation w 1 d 1 + w 2 d 2 + w 3 d 3 = θ A linear classifier in 3D is a plane described by the equation β 1 x 1 + β 2 x 2 + β 3 x 3 = β 0 Example for a 3D linear classifier Example for a 3D linear classifier Schütze: Support vector machines 17 / 55 Machine Learning: Jordan Boyd-Graber UMD Introduction to Machine Learning 11 / 29

Linear Classifiers Recap Vector space classification Linear classifiers Support Vector Machines Discussion A linear classifier in 3D Alinearclassifierin3D A linear classifier in 3D is a plane described by the Alinearclassifierin3Dis aplanedescribedbythe equation equation w 1 d 1 + w 2 d 2 + w 3 d 3 = θ β 1 x 1 + β 2 x 2 + β 3 x 3 = β 0 Example for a 3D linear classifier Points (d 1 d 2 d 3 )with w 1 d 1 + w 2 dclassifier 2 + w 3 d 3 θ are in the class c. Example for a 3D linear Points (x1 x 2 x 3 ) with β 1 x 1 + β 2 x 2 + β 3 x 3 β 0 are in the class c. Schütze: Support vector machines 17 / 55 Machine Learning: Jordan Boyd-Graber UMD Introduction to Machine Learning 11 / 29

Linear Classifiers Recap Vector space classification Linear classifiers Support Vector Machines Discussion A linear Alinearclassifierin3D classifier in 3D Alinearclassifierin3Dis aplanedescribedbythe equation w 1 d 1 + w 2 d 2 + w 3 d 3 = θ Example for a 3D linear classifier Points (d 1 d 2 d 3 )with w 1 d 1 + w 2 d 2 + w 3 d 3 θ are in the class c. Points (d 1 d 2 d 3 )with w 1 d 1 + w 2 d 2 + w 3 d 3 < θ are in the complement class c. Schütze: Support vector machines 17 / 55 A linear classifier in 3D is a plane described by the equation β 1 x 1 + β 2 x 2 + β 3 x 3 = β 0 Example for a 3D linear classifier Points (x1 x 2 x 3 ) with β 1 x 1 + β 2 x 2 + β 3 x 3 β 0 are in the class c. Points (x1 x 2 x 3 ) with β 1 x 1 + β 2 x 2 + β 3 x 3 < β 0 are in the complement class c. Machine Learning: Jordan Boyd-Graber UMD Introduction to Machine Learning 11 / 29

Linear Classifiers Naive Bayes and Logistic Regression as linear classifiers Multinomial Naive Bayes is a linear classifier (in log space) defined by: M β i x i = β 0 i=1 where β i = log[ˆp(ti c)/ˆp(ti c)], x i = number of occurrences of t i in d, and β 0 = log[ˆp(c)/ˆp( c)]. Here, the index i, 1 i M, refers to terms of the vocabulary. Logistic regression is the same (we only put it into the logistic function to turn it into a probability). Machine Learning: Jordan Boyd-Graber UMD Introduction to Machine Learning 12 / 29

Linear Classifiers Naive Bayes and Logistic Regression as linear classifiers Multinomial Naive Bayes is a linear classifier (in log space) defined by: M β i x i = β 0 i=1 where β i = log[ˆp(ti c)/ˆp(ti c)], x i = number of occurrences of t i in d, and β 0 = log[ˆp(c)/ˆp( c)]. Here, the index i, 1 i M, refers to terms of the vocabulary. Logistic regression is the same (we only put it into the logistic function to turn it into a probability). Takeway Naïve Bayes, logistic regression and SVM are all linear methods. They choose their hyperplanes based on different objectives: joint likelihood (NB), conditional likelihood (LR), and the margin (SVM). Machine Learning: Jordan Boyd-Graber UMD Introduction to Machine Learning 12 / 29

Linear Classifiers Which hyperplane? Machine Learning: Jordan Boyd-Graber UMD Introduction to Machine Learning 13 / 29

Linear Classifiers Which hyperplane? For linearly separable training sets: there are infinitely many separating hyperplanes. They all separate the training set perfectly...... but they behave differently on test data. Error rates on new data are low for some, high for others. How do we find a low-error separator? Machine Learning: Jordan Boyd-Graber UMD Introduction to Machine Learning 14 / 29

Support Vector Machines Support vector machines Machine-learning research in the last two decades has improved classifier effectiveness. New generation of state-of-the-art classifiers: support vector machines (SVMs), boosted decision trees, regularized logistic regression, neural networks, and random forests Applications to IR problems, particularly text classification SVMs: A kind of large-margin classifier Vector space based machine-learning method aiming to find a decision boundary between two classes that is maximally far from any point in the training data (possibly discounting some points as outliers or noise) Machine Learning: Jordan Boyd-Graber UMD Introduction to Machine Learning 15 / 29

Support Vector Machines Support Vector Machines Support Vector Machines 2-class training data 2-class training data Schütze: Support vector machines 29 / 55 Machine Learning: Jordan Boyd-Graber UMD Introduction to Machine Learning 16 / 29

pport Vector Machines Support Vector Machines 2-class training data Support Vector Machines decision boundary linear separator 2-class training data decision boundary linear separator ütze: Support vector machines 29 / 55 Machine Learning: Jordan Boyd-Graber UMD Introduction to Machine Learning 16 / 29

pport Vector Machines Support Vector Machines Support Vector Machines 2-class training data decision boundary linear separator 2-class training data criterion: decision being boundary maximally linear separator far away from criterion: any data being point maximally determinesfar away classifier from any margin data point determines classifier margin Margin is maximized tze: Support vector machines 29 / 55 Machine Learning: Jordan Boyd-Graber UMD Introduction to Machine Learning 16 / 29

Support Vector Machines Recap Vector space classification Linear classifiers Support Vector Machines Discussion Support Vector Machines Why maximize the margin? Points near decision surface uncertain classification decisions 2-class (50% training either data way). decision Aclassifierwithalarge boundary margin makes no low linear certainty separator classification criterion: decisions. being Gives classification maximally far away safety margin w.r.t slight from any errors data in measurement point or determines doc. variation classifier margin linear separator position defined by support vectors Maximum margin decision hyperplane Support vectors Margin is maximized Schütze: Support vector machines 30 / 55 Machine Learning: Jordan Boyd-Graber UMD Introduction to Machine Learning 16 / 29

Support Vector Machines Recap Vector space classification Linear classifiers Support Vector Machines Discussion Why maximize the margin? Why maximize the margin? Points near decision surface uncertain classification decisions Points (50% near either decision way). surfaceaclassifierwithalarge uncertain classification margin makes decisions no low certainty classification A classifier decisions. with a large margingives is always classification safety margin w.r.t slight confident errors in measurement or Gives classification doc. variation safety margin (measurement or variation) Maximum margin decision hyperplane Support vectors Margin is maximized Schütze: Support vector machines 30 / 55 Machine Learning: Jordan Boyd-Graber UMD Introduction to Machine Learning 17 / 29

Support Vector Machines Why maximize the margin? SVM classifier: large margin around decision boundary compare to decision hyperplane: place fat separator between classes unique solution decreased memory capacity increased ability to correctly generalize to test data Machine Learning: Jordan Boyd-Graber UMD Introduction to Machine Learning 18 / 29

Formulation Equation Equation of a hyperplane w x i + b = 0 (1) Distance of a point to hyperplane w x i + b w (2) The margin ρ is given by w x i + b ρ min = 1 (x,y) S w w (3) Machine Learning: Jordan Boyd-Graber UMD Introduction to Machine Learning 19 / 29

Formulation Equation Equation of a hyperplane w x i + b = 0 (1) Distance of a point to hyperplane w x i + b w (2) The margin ρ is given by w x i + b ρ min = 1 (x,y) S w w (3) This is because for any point on the marginal hyperplane, w x + b = ±1 Machine Learning: Jordan Boyd-Graber UMD Introduction to Machine Learning 19 / 29

Formulation Optimization Problem We want to find a weight vector w and bias b that optimize subject to y i ( w x i + b) 1, i [1,m]. 1 min w,b 2 w 2 (4) Machine Learning: Jordan Boyd-Graber UMD Introduction to Machine Learning 20 / 29

Formulation Optimization Problem We want to find a weight vector w and bias b that optimize subject to y i ( w x i + b) 1, i [1,m]. Next week: algorithm 1 min w,b 2 w 2 (4) Machine Learning: Jordan Boyd-Graber UMD Introduction to Machine Learning 20 / 29

Theoretical Guarantees Three Proofs that Suggest SVMs will Work Leave-one-out error VC Dimension Margin analysis Machine Learning: Jordan Boyd-Graber UMD Introduction to Machine Learning 21 / 29

Theoretical Guarantees Leave One Out Error (sketch) Leave one out error is the error by using one point as your test set (averaged over all such points). ˆR LOO = 1 m m 1 h s {xi } y i i=1 (5) This serves as an unbiased estimate of generalization error for samples of size m 1: S D m ˆRLOO = S D m 1 [R(h S )] (6) Machine Learning: Jordan Boyd-Graber UMD Introduction to Machine Learning 22 / 29

Theoretical Guarantees Leave One Out Error (sketch) Let h S be the hypothesis returned by SVMs for a separable sample S, and let N SV (S) be the number of support vectors that define h S. S D m [R(h s )] S D m+1 Consider the held out error for x i. NSV (S) m + 1 If xi was not a support vector, the answer doesn t change. If xi was a support vector, it could change the answer; this is when we can have an error. There are N SV (S) support vectors and thus N SV (S) possible errors. (7) Machine Learning: Jordan Boyd-Graber UMD Introduction to Machine Learning 23 / 29

Theoretical Guarantees VC Dimension Argument Remember discussion VC dimension for d-dimensional hyperplanes? That applies here: R(h) ˆR(h) + 2(d + 1)log m ε d+1 log 1 δ + 2m (8) Machine Learning: Jordan Boyd-Graber UMD Introduction to Machine Learning 24 / 29

Theoretical Guarantees VC Dimension Argument Remember discussion VC dimension for d-dimensional hyperplanes? That applies here: R(h) ˆR(h) + 2(d + 1)log m ε d+1 But this is useless when d is large (e.g. for text). log 1 δ + 2m (8) Machine Learning: Jordan Boyd-Graber UMD Introduction to Machine Learning 24 / 29

Theoretical Guarantees Margin Theory Machine Learning: Jordan Boyd-Graber UMD Introduction to Machine Learning 25 / 29

Theoretical Guarantees Margin Loss Function To see where SVMs really shine, consider the margin loss ρ: 0 if ρ x Φ ρ (x) = 1 x ρ if 0 x ρ 1 if x 0 (9) Machine Learning: Jordan Boyd-Graber UMD Introduction to Machine Learning 26 / 29

Theoretical Guarantees Margin Loss Function To see where SVMs really shine, consider the margin loss ρ: 0 if ρ x Φ ρ (x) = 1 x ρ if 0 x ρ 1 if x 0 (9) The empirical margin loss of a hypothesis h is ˆR ρ (h) = 1 m m Φ ρ (y i h(x i )) 1 m i=1 m 1[y i h(x i ) ρ] (10) i=1 Machine Learning: Jordan Boyd-Graber UMD Introduction to Machine Learning 26 / 29

Theoretical Guarantees Margin Loss Function To see where SVMs really shine, consider the margin loss ρ: 0 if ρ x Φ ρ (x) = 1 x ρ if 0 x ρ 1 if x 0 (9) The empirical margin loss of a hypothesis h is ˆR ρ (h) = 1 m m Φ ρ (y i h(x i )) 1 m i=1 m 1[y i h(x i ) ρ] (10) The fraction of the points in the training sample S that have been misclassified or classified with confidence less than ρ. i=1 Machine Learning: Jordan Boyd-Graber UMD Introduction to Machine Learning 26 / 29

Theoretical Guarantees Generalization For linear classifiers H = {x w x : w Λ} and data X {x : x r}. Fix ρ > 0 then with probability at least 1 δ, for any h H, r 2Λ R(h) ˆRρ (h) + 2 log 1 2 ρ 2 m + δ 2m (11) Data-dependent: must be separable with a margin Fortunately, many data do have good margin properties SVMs can find good classifiers in those instances Machine Learning: Jordan Boyd-Graber UMD Introduction to Machine Learning 27 / 29

Recap Choosing what kind of classifier to use When building a text classifier, first question: how much training data is there currently available? None? Very little? A fair amount? A huge amount Machine Learning: Jordan Boyd-Graber UMD Introduction to Machine Learning 28 / 29

Recap Choosing what kind of classifier to use When building a text classifier, first question: how much training data is there currently available? None? Hand write rules or use active learning Very little? A fair amount? A huge amount Machine Learning: Jordan Boyd-Graber UMD Introduction to Machine Learning 28 / 29

Recap Choosing what kind of classifier to use When building a text classifier, first question: how much training data is there currently available? None? Hand write rules or use active learning Very little? Naïve Bayes A fair amount? A huge amount Machine Learning: Jordan Boyd-Graber UMD Introduction to Machine Learning 28 / 29

Recap Choosing what kind of classifier to use When building a text classifier, first question: how much training data is there currently available? None? Hand write rules or use active learning Very little? Naïve Bayes A fair amount? SVM A huge amount Machine Learning: Jordan Boyd-Graber UMD Introduction to Machine Learning 28 / 29

Recap Choosing what kind of classifier to use When building a text classifier, first question: how much training data is there currently available? None? Hand write rules or use active learning Very little? Naïve Bayes A fair amount? SVM A huge amount Doesn t matter, use whatever works Machine Learning: Jordan Boyd-Graber UMD Introduction to Machine Learning 28 / 29

Recap SVM extensions: What s next Finding solutions Slack variables: not perfect line Kernels: different geometries 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 Machine Learning: Jordan Boyd-Graber UMD Introduction to Machine Learning 29 / 29