Support Vector Machines. CAP 5610: Machine Learning Instructor: Guo-Jun QI

Transcription

1 Support Vector Machines CAP 5610: Machine Learning Instructor: Guo-Jun QI 1

2 Linear Classifier Naive Bayes Assume each attribute is drawn from Gaussian distribution with the same variance Generative model: estimate the mean and variance with closed-form solution Logistic regression Directly maximizing the log likelihood to fit the model into the training data Discriminative model: no closed-form solution, a gradient ascent method is used. 2

3 Drawback Lacking of a geometric intuition to explain what s a good linear classifier in high dimensional space. 3

4 SVM Supervised learning methods used for Classification Regression A special property: simultaneously minimize the classification error maximize the geometric margin maximum margin classifier Excellent theory and good performance 4

5 Outline Linear SVM hard margin Linear SVM soft margin Non-linear SVM Application 5

6 Outline Linear SVM hard margin Linear SVM soft margin Non-linear SVM Application 6

7 Linear Classifiers Label y: denotes +1 denotes -1 x w x + b>0 Parameters f f(x,w,b) = sign(w x + b) y How would you classify this data? w x + b<0

8 Linear Classifiers x Parameters f y est denotes +1 denotes -1 f(x,w,b) = sign(w x + b) How would you classify this data?

9 Linear Classifiers x Parameters f y est denotes +1 denotes -1 f(x,w,b) = sign(w x + b) How would you classify this data?

10 Linear Classifiers x Parameters f y est denotes +1 denotes -1 f(x,w,b) = sign(w x + b) Any of these would be fine....but which is best?

11 Linear Classifiers x Parameters f y est denotes +1 denotes -1 f(x,w,b) = sign(w x + b) How would you classify this data? Misclassified to +1 class

12 Classifier Margin x Parameters f y est denotes +1 denotes -1 f(x,w,b) = sign(w x + b) Define the margin of a linear classifier as the width that the boundary could be increased by before hitting a data point.

13 Maximum Margin denotes +1 denotes -1 Support Vectors are those datapoints that the margin pushes up against x f y est 1. Maximizing the margin makes sense according to intuition f(x,w,b) = sign(w x + b) 2. Implies that only support vectors are important; other training The maximum examples can be discarded without affecting the training result. Linear SVM Parameters margin linear classifier is the linear classifier with the maximum margin. This is the simplest kind of SVM (Called an LSVM)

14 Maximum Margin x Parameters f y est denotes +1 denotes -1 Keeping only support vectors will not change the maximum margin classifier. Robust to the small changes (noises) in non-support vectors Linear SVM f(x,w,b) = sign(w x + b) The maximum margin linear classifier is the linear classifier with the maximum margin. This is the simplest kind of SVM (Called an LSVM)

15 Basics to SVM math w/ w : Perpendicular to line wx+b=0 Unit length Margin of two parallel lines is w/ w w 2 x1 x2 w b1 b w x 1 where wx wx b 1 1 b w( x x ) b b w/ w x 2 15

16 Linear SVM Mathematically x + x - Decision rule: Positive examples: w. x + + b > +1 Negative examples: w. x - + b < -1 Subtracting two equations: w. (x + -x - ) = 2

17 Linear SVM Mathematically x + M=Margin Width x - What we know: w. x + + b = +1 w. x - + b = -1 w. (x + -x - ) = 2 M ( x x w ) w 2 w

18 Linear SVM Mathematically Goal: 1) Correctly classify all training data 2) Maximize the Margin same as minimize if y i = +1 if y i = -1 We can formulate a Quadratic Optimization Problem and solve for w and b Minimize subject to wx i b 1 wx i b 1 y i ( wx b) 1 ( w) y i i 1 2 t w w ( wx b) 1 i for all i M 2 1 w w t w 2 i

19 Solving the Optimization Problem Need to optimize a quadratic function subject to linear constraints. Use Lagrange multiplier. α i is associated with every constraint : y ( wx b) 1, dual problem i i Find α 1 α N such that Q(α) =Σα i - ½ΣΣα i α j y i y j x it x j is maximized and (1) Σα i y i = 0 (2) α i 0 for all α i Refer: Christopher J. C. Burges: A Tutorial on Support Vector Machines for Pattern Recognition, Data Mining and Knowledge Discovery, 1998

20 The Optimization Problem Solution The solution has the form: w =Σα i y i x i b= y k - w T x k for any x k such that α k 0 α i must satisfy Karush-Kuhn-Tucker conditions: α i [y i (w T x i +b)-1]=0, for any i If α i >0, y i (w T x i +b)-1=0, x i is on the margin If y i (w T x i +b)>1, α i =0 Each non-zero α i indicates that corresponding x i is a support vector.

21 Maximum Margin denotes +1 denotes -1 w, b depends only on Support Vectors via active constraints y i (w T x k +b)-1=0

22 The Optimization Problem Solution To classify the new test point x, we use f(x) = wx + b =Σα i y i x it x + b Find α 1 α N such that Q(α) =Σα i - ½ΣΣα i α j y i y j x it x j is maximized and (1) Σα i y i = 0 (2) α i 0 for all α i