Support Vector Machines

Transcription

1 Supervised Learning Methods -nearest-neighbors (-) Decision trees eural netors Support vector machines (SVM) Support Vector Machines Chapter 18.9 and the optional paper Support vector machines by M. Hearst, ed., 1998 Acnoledgments: These slides combine and modify ones provided by Andre Moore (CMU), Carla Gomes (Cornell), Mingyue Tan (UBC), Jerry Zhu (Wisconsin), Glenn Fung (Wisconsin), and Olvi Mangasarian (Wisconsin) What are Support Vector Machines (SVMs) Used For? Classification Regression and data-fitting Supervised and unsupervised learning Lae Mendota, Madison, WI Identify areas of land cover (land, ice, ater, sno) in a scene To methods: Scientist manually-derived Support Vector Machine (SVM) Classifier Expert Derived SVM cloud 45.7% 58.5% ice 60.1% 80.4% Lae Mendota, Wisconsin land 93.6% 94.0% sno 63.5% 71.6% ater 84.% 89.1% unclassified 45.7% Courtesy of Steve Chien of ASA/JPL Visible Image Expert Labeled Expert Automated SVM Derived Ratio 1

2 Linear Classifiers x f y Linear Classifiers (aa Linear Discriminant Functions) Ho ould you classify this data? Definition: A function that is a linear combination of the components of the input (column vector) x: f (x) = d j=1 j x j + b = T x + b here is the eight (column vector) and b is the bias A -class classifier then uses the rule: Decide class c 1 if f(x) 0 and class c if f(x) < 0 or, equivalently, decide c 1 if T x -b and c otherise Planar decision surface in d dimensions is parameterized by (, b) is the plane s normal vector b is the distance from the origin Linear Classifiers x f y f(x,, b) = sign( T x + b) Ho ould you classify this data? b

3 Linear Classifiers x f y Linear Classifiers x f y f(x,, b) = sign( T x + b) f(x,, b) = sign( T x + b) Ho ould you classify this data? Ho ould you classify this data? Linear Classifiers x f y Classifier Margin x f y f(x,, b) = sign( T x + b) Any of these ould be fine but hich is best? f(x,, b) = sign( T x + b) Define the margin of a linear classifier as the idth that the decision boundary could be expanded before hitting a data point 3

4 Maximum Margin x f y Maximum Margin x f y Linear SVM f(x,, b) = sign( T x + b) The maximum margin linear classifier is the linear classifier ith the maximum margin This is the simplest ind of SVM (Called an LSVM) Support Vectors are those data points that the margin pushes against Linear SVM f(x,, b) = sign( T x + b) The maximum margin linear classifier is the linear classifier ith the maximum margin This is the simplest ind of SVM (Called an LSVM) Why the Maximum Margin? Support Vectors are those data points that the margin pushes against 1. Intuitively this feels safest. If e ve made f(x,,b) a small = sign(. error in xthe - b) location of the boundary (it s been jolted in its perpendicular The maximum direction) this gives us least margin chance of linear causing a misclassification classifier is the 3. Robus to outliers linear since the classifier model is immune to change/removal ith the, of um, any non-support-vector data points maximum margin. 4. There s some theory (using VC dimension ) that This related is the to (but not the same as) the simplest proposition ind that of this is a good thing SVM (Called an 5. Empirically it ors LSVM) very ell Specifying a Line and Margin Plus-Plane Ho do e represent this mathematically? in d input dimensions? An example x = (x 1,, x d ) T Classifier Boundary Minus-Plane 4

5 Specifying a Line and Margin Plus-plane = T x + b = +1 Minus-plane = T x + b = -1 Plus-Plane Classify as +1 if T x + b 1-1 if T x + b -1 Classifier Boundary Minus-Plane Weight vector: = ( 1,, d ) T Bias or threshold: b? if -1 < T x + b < 1 The dot product T x = d j x j j=1 is a scalar: x s projection onto Computing the Margin T x+b=1 T x+b=0 T x+b=-1 Plus-plane = T x + b = +1 Minus-plane = T x + b = -1 M = Margin (idth) Ho do e compute M in terms of and b? Claim: The vector is perpendicular to the Plus-Plane and the Minus-Plane Computing the Margin M = Margin Ho do e compute M in terms of and b? Plus-plane = T x + b = +1 Minus-plane = T x + b = -1 The vector is perpendicular to the Plus-Plane Let x be any point on the Minus-Plane Let x + be the closest Plus-Plane-point to x x + x Any location in R m d :; not necessarily a datapoint Computing the Margin M = Margin Ho do e compute M in terms of and b? Plus-plane = T x + b = +1 Minus-plane = T x + b = -1 The vector is perpendicular to the Plus-Plane Let x be any point on the Minus-Plane Let x + be the closest Plus-Plane-point to x Claim: x + = x + λ for some value of λ x + x 5

6 Computing the Margin The line from x to x + is Ho do e compute x perpendicular to the planes M in terms of So to and get b? from x to x + travel some distance in Plus-plane = T x + b = +1 direction Minus-plane = T x + b = -1 The vector is perpendicular to the Plus-Plane Let x be any point on the Minus-Plane Let x + be the closest Plus-Plane-point to x Claim: x + = x + λ for some value of λ Why? x + M = Margin What e no: T x + + b = +1 T x - + b = -1 x + = x + λ x + - x = M Computing the Margin x + x M = Margin What e no: T x + + b = +1 T x + b = -1 x + = x + λ x + - x = M It s no easy to get M in terms of and b Computing the Margin x + x M = Margin T (x + λ) + b = 1 T x + b + λ T = λ T = 1 l = T What e no: T x + + b = +1 T x + b = -1 x + = x + λ x + - x = M Computing the Margin x + x M = Margin = λ = T M = x + - x = λ = λ = λ T = T T = T T = = M, margin size 6

7 Learning the Maximum Margin Classifier x + M = Margin = T Given a guess of and b, e can 1. Compute hether all data points in the correct half-planes. Compute the margin So no e just need to rite a program to search the space of s and b s to find the idest margin that correctly classifies all the data points Ho? x SVM as Constrained Optimization Unnons:, b Objective function: maximize the margin: M = / Equivalent to minimizing or = T training points: (x, y ), y = +1 or -1 Subject to each training point correctly classified, i.e., subject to y ( T x + b) 1 for all This is a quadratic optimization problem (QP), hich can be solved efficiently constraints Classification ith SVMs Given a ne point x, e can classify it by Computing score: T x + b Deciding class based on hether < 0 or > 0 If desired, can set confidence threshold t Score > t : yes Score < -t : no Else: don t no Sec SVMs: More than To Classes SVMs can only handle to-class problems -class problem: Split the tas into binary tass and learn SVMs: Class 1 vs. the rest (classes ) Class vs. the rest (classes 1, 3 ) Class vs. the rest Pic the class that puts the point farthest into its positive region 7

8 SVMs: on Linearly-Separable Data What if the data are not linearly separable? from Statniov et al. SVMs: on Linearly-Separable Data To approaches: Allo a fe points on the rong side (slac variables) Map data to a higher dimensional space, and do linear classification there (ernel tric) on Linearly-Separable Data Approach 1: Allo a fe points on the rong side (slac variables) Soft Margin Classification 8

9 What Should We Do? What Should We Do? Idea: Minimize + C (# train errors) Tradeoff penalty parameter There s a serious practical problem ith this approach What Should We Do? What Should We Do? Idea: Minimize + C (# train errors) Tradeoff penalty parameter Minimize + C (distance of all misclassified points to their correct place) Can t be expressed as a Quadratic Programming problem. So solving it may be too slo. (Also, doesn t distinguish beteen disastrous errors and near misses) There s a serious practical problem ith this approach 9

10 Choosing C, the Penalty Factor Choosing C Critical to choose a good value for the constant penalty parameter, C, because C too big means very similar to LSVM, e have a high penalty for non-separable points, and e may use many support vectors and overfit C too small means e allo many misclassifications in the training data and e may underfit from Statniov et al. Learning Maximum Margin ith oise M = T Given guess of, b, e can 1. Compute sum of distances of points to their correct s. Compute the margin idth Assume examples, each (x, y ) here y = +1 / -1 Learning Maximum Margin ith oise ε ε 11 ε 7 M = slac variables T Given guess of, b, e can 1. Compute sum of distances of points to their correct s. Compute the margin idth Assume examples, each (x, y ) here y = +1 / -1 What should our optimization criterion be? What should our optimization criterion be? Minimize 1 T + C ε =1 Ho many constraints ill e have? What should they be? y ( T x + b) 1-ε for all ote: ε = 0 for points in correct 10

11 Learning Maximum Margin ith oise d = # input M = dimensions Given guess of, b e can ε 11 ε 1. Compute sum of distances of points to their correct Our original (noiseless data) QP had d +1 s variables: 1,, d, and b e 7. Compute the margin idth Our ne (noisy data) QP has d +1+ Assume R datapoints, each variables: 1,, (x,y d, b, ε ) here, ε y 1, ε = +/- 1 Learning Maximum Margin ith oise M = Given guess of, b e can ε Compute sum of distances T of points to their correct s ε 7. Compute the margin idth Assume examples, each slac variables (x, y ) here y = +1 / -1 ε What should our optimization criterion be? Minimize 1 T + C ε =1 Ho many constraints ill e = # examples have? What should they be? T x + b 1- ε if y = +1 T x + b -1+ε if y = -1 What should our optimization criterion be? Minimize 1 T + C ε =1 Ho many constraints ill e have? What should they be? T x + b +1 - ε if y = +1 T x + b -1 + ε if y = -1 ε 0 for all on Linearly-Separable Data Approach : Map data to a higher dimensional space, and then do linear classification there (called the ernel tric) Suppose e re in 1 Dimension What ould SVMs do ith this data? x=0 11

12 Suppose e re in 1 Dimension Harder 1D Dataset: ot Linearly-Separable What can be done about this? x=0 Positive plane egative plane x=0 Harder 1D Dataset The ernel Tric: Preprocess the data, mapping x into a higher dimensional space, z = Φ(x) For example: x=0 F( x ) = ( x, x ) Here, Φ maps data from 1D to D In general, map from d-dimensional input space to -dimensional z space The data is linearly separable in the ne space, so use a linear SVM in the ne space Harder 1D Dataset The ernel Tric: Preprocess the data, mapping x into a higher dimensional space, z = Φ(x) x=0 F( x ) = ( x, x ) T Φ(x) + b = +1 1

13 Another Example F( x 1, x) = ( x1, x, x1 + x ) Another Example Project examples into some higher dimensional space here the data is linearly separable, defined by z = Φ(x) Project examples into some higher dimensional space here the CS 540, University of Wisconsin-Madison, C. R. Dyer data is linearly separable, defined by z = Φ(x) Algorithm 1. Pic a Φ function. Map each training example, x, into the ne higher-dimensional space using z = Φ(x) 3. Solve the optimization problem using the nonlinearly transformed training examples, z, to obtain a Linear SVM (ith or ithout using the slac variable formulation) defined by and b 4. Classify a test example, e, by computing: sign( T Φ(e) + b) Improving Efficiency Time complexity of the original optimization formulation depends on the dimensionality,, of z ( >> d) We can convert the optimization problem into an equivalent form, called the Dual Form, ith time complexity O( 3 ) that depends on, the number of training examples, not Dual Form ill also allo us to rerite the mapping functions in Φ in terms of ernel functions instead 13

14 Maximize Dual Optimization Problem åα + ååα αq here Q ( T l = y yl x. x ) l l l = 1 = 1 l= 1 Maximize Dot product of Dual Optimization Problem åα + ååα αq here Q ( T l = y yl x. x ) l l l = 1 = 1 l= 1 to examples subject to these constraints: 0 α C " å = 1 α y = 0 subject to these constraints: 0 α C " å = 1 α y = 0 Then define: = åα y x = 1 b = y (1 - ε ) - x. here = arg max α Then classify ith: f(x,,b) = sign( T x - b) examples: (x, y ) here y = +1 / -1 Then define: = åα y x = 1 b = y (1 - ε ) - x. here = arg max α e variables; Examples ith α > 0 ill be the support vectors Then classify ith: f(x,,b) = sign( T x - b) examples: (x, y ) here y = +1 / -1 Algorithm Compute x matrix Q by computing y i y j (x it x j ) beteen all pairs of training data points Solve the optimization problem to compute α i for i = 1,, Each non-zero α i indicates that example x i is a support vector Compute and b Then classify test example x ith: f(x) = sign( T x b) Dual Optimization Problem (After Mapping) 1 α - α α Q here Q l = y y l (Φ(x ) T Φ(x l )) Maximize Subject to these constraints: Then define: = å å α y s.t. α > 0 åå = 1 = 1 l= 1 0 α C " Φ ( x b = y (1 - ε ) - x. here = arg max α l ) l å = 1 Then classify ith: α y = 0 f(x,,b) = sign( T Φ(x) - b) examples: (x, y ) here y = +1 / -1 Copyright 001, 003, Andre W. Moore 14

15 Dual Optimizzation Problem (After Mapping) 1 α - α α Q here Q l = y y l (Φ(x ) T Φ(x l )) Maximize Subject to these constraints: Then define: = å å α y s.t. α > 0 åå = 1 = 1 l= 1 0 α C " Φ ( x b = y (1 - ε ) - x. here = arg max α l ) l å dot products to compute this matrix = 1 Then classify ith: α y = 0 f(x,,b) = sign( T Φ(x) - b) examples: (x, y ) here y = +1 / -1 Dual formulation of the optimization problem depends on the input data only in dot products of the form: Φ(x ) T i Φ(x ) j here x i and x j are to examples We can compute these dot products efficiently for certain types of Φ s here (x, i x ) j = Φ(x ) T i Φ(x ) j Example: F( x) = ( x1, x1x, x ) Φ(x i ) T Φ(x j ) = (x it x j ) = (x i, x j ) Since the data only appears as dot products, e do not need to map the data to higher dimensional space (using Φ(x) ) because e can use the ernel function instead Copyright 001, 003, Andre W. Moore F( x) = ( x1, x1x, x Φ( x i ) T Φ( x j ) = x i1 x j1 + x i1 x i x j1 x j + x i x j = x i1 x j1 + x i1 x i x j1 x j + x i x j ( ) ( ) = x i1 x j1 + x i x j = x i T x j ) ernel Functions A ernel is a dot product in some feature space A ernel function is a function that can be applied to pairs of input data points to evaluate dot products in some corresponding (possibly infinite dimensional) feature space We do not need to compute Φ explicitly 15

16 What s Special about a ernel? Say 1 example is (in D) is: s = (s 1, s ) We decide to use a particular mapping into 6D space: Φ(s) T = (s 1, s, s 1 s, s 1, s, 1) Let another example be t = (t 1, t ) Then, Φ(s) T Φ(t) = s 1 1 t + s t + s 1 s 1 t t + s 1 1 t + s t + 1 = (s 1 1 t + s t + 1) = (s T t +1) So, define the ernel function to be (s, t) = (s T t +1) We save computation by using = Φ(s) T Φ(t) Some Commonly Used ernels Linear ernel: (x i, x j ) = x i T x j Quadratic ernel: (x i, x j ) = (x i T x j +1) Polynomial of degree d ernel: (x i, x j ) = (x i T x j +1) d Radial-Basis Function (Gaussian) ernel: (x i, x j ) = exp( x i -x j / σ ) Many possible ernels; picing a good one is tricy Hacing ith SVMs: create various ernels, hope their space Φ is meaningful, plug them into SVM, pic one ith good classification accuracy ernel usually combined ith slac variables because no guarantee of linear separability in ne space Algorithm Compute x matrix Q by computing y i y j (x i,x j ) beteen all pairs of training points Solve optimization problem to compute α i for i = 1,, Each non-zero α i indicates that example x i is a support vector Compute and b Classify test example x using: f(x) = sign( T x b) Applications of SVMs n Bioinformatics n Machine Vision n Text Categorization n Raning (e.g., Google searches) n Handritten Character Recognition n Time series analysis à Lots of very successful applications! 16

17 Handritten Digit Recognition Example Application: The Federalist Papers Dispute Written in by Alexander Hamilton, John Jay, and James Madison to persuade the citizens of e Yor to ratify the U.S. Constitution Papers consisted of short essays, 900 to 3500 ords in length Authorship of 1 of those papers have been in dispute ( Madison or Hamilton); these papers are referred to as the disputed Federalist papers Description of the Data For every paper: Machine readable text as created using a scanner Computed relative frequencies of 70 ords that Mosteller-Wallace identified as good candidates for author-attribution studies Each document is represented as a vector containing the 70 real numbers corresponding to the 70 ord frequencies The dataset consists of 118 papers: 50 Madison papers 56 Hamilton papers 1 disputed papers Bag of ords 70-Word Dictionary 17

18 Feature Selection for Classifying the Disputed Federalist Papers Apply the SVM algorithm for feature selection to: Train on the 106 Federalist papers ith non authors Find a classification hyperplane that uses as fe ords as possible Use the hyperplane to classify the 1 disputed papers Hyperplane Classifier Using 3 Words A hyperplane depending on three ords as found: 0.537to upon +.953ould = All disputed papers ended up on the Madison side of the plane Results: 3D Plot of Hyperplane Summary Learning linear functions Pic separating hyperplane that maximizes margin Separating plane defined in terms of support vectors (small number of training data points) only Learning non-linear functions Project examples into higher dimensional space Use ernel functions for efficiency Generally avoids overfitting problem Global optimization method; no local optima Can be expensive to apply, especially for multi-class problems Biggest Drabac: The choice of ernel function There is no set-in-stone theory for choosing a ernel function for any given problem Once a ernel function is chosen, there is only OE modifiable parameter, the error penalty C 18

19 Softare A list of SVM implementations can be found at Some implementations (such as LIBSVM) can handle multi-class classification SVMLight is one of the earliest and most frequently used implementations of SVMs Several Matlab toolboxes for SVMs are available 19