Support Vector Machine (SVM) and Kernel Methods CE-717: Machine Learning Sharif University of Technology Fall 2016 Soleymani
Outline Margin concept Hard-Margin SVM Soft-Margin SVM Dual Problems of Hard-Margin SVM and Soft-Margin SVM onlinear SVM Kernel trick Kernel methods 2
Margin Which line is better to select as the boundary to provide more generalization capability? Larger margin provides better generalization to unseen data x 2 Margin for a hyperplane that separates samples of two linearly separable classes is: The smallest distance between the decision boundary and any of the training samples x 1 3
What is better linear separation Linearly separable data Which line is better? Why the bigger margin? 4
Maximum margin SVM finds the solution with maximum margin Solution: a hyperplane that is farthest from all training samples x 2 x 2 x 1 Larger margin x 1 The hyperplane with the largest margin has equal distances to the nearest sample of both classes 5
Finding w with large margin Two preliminaries: Pull out w 0 w is w 1,, w d ormalize w, w 0 Let x (n) be the nearest point to the plane w T x (n) + w 0 = 1 w T x + w 0 = 0 We have no x 0 6
Distance between an x (n) and the plane distance = wt x (n) + w 0 w x (n) distance w w 7
The optimization problem 2 max w,w 0 w s. t. min,, wt x n + w 0 = 1 From all the hyperplanes that correctly classify data otice: w T x n + w 0 = y n w T x n + w 0 1 min w,w 0 2 w 2 s. t. y n w T x n + w 0 1 n = 1,, 8
Hard-margin SVM: Optimization problem 2 max w,w 0 w s. t. w T x n + w 0 1, n = 1,, w T x + w 0 = 0 x 2 1 w Margin: 2 w w w T x + w 0 = 1 w T x + w 0 = 1 9 x 1
Hard-margin SVM: Optimization problem 2 max w,w 0 w s. t. w T x n + w 0 1 y n = 1 w T x n + w 0 1 y n = 1 w T x + w 0 = 0 x 2 1 w Margin: 2 w w w T x + w 0 = 1 w T x + w 0 = 1 10 x 1
Hard-margin SVM: Optimization problem We can equivalently optimize: 1 min w,w 0 2 wt w s. t. y n w T x n + w 0 1 n = 1,, It is a convex Quadratic Programming (QP) problem There are computationally efficient packages to solve it. It has a global minimum (if any). 11
Quadratic programming min 2 xt Qx + c T x s. t. Ax b Ex = d x 1 12
Dual formulation of the SVM We are going to introduce the dual SVM problem which is equivalent to the original primal problem. The dual problem: is often easier gives us further insights into the optimal hyperplane enable us to exploit the kernel trick 13
Optimization: Lagrangian multipliers p = min f(x) x s. t. g i x 0 i = 1,, m h i x = 0 i = 1,, p Lagrangian multipliers m p L x, α, λ = f x + α i g i x + λ i h i x i=1 i=1 max L x, α, λ = α i 0, λ i any g i x > 0 any h i x 0 f x otherwise p = min x max α i 0, λ i L x, α, λ α = α 1,, α m λ = [λ 1,, λ p ] 14
Optimization: Dual problem In general, we have: max min x y h(x, y) min y max x h(x, y) Primal problem: p = min x max α i 0, λ i L x, α, λ Dual problem: d = max α i 0, λ i min x L x, α, λ Obtained by swapping the order of min and max d p When the original problem is convex (f and g are convex functions and h is affine), we have strong duality d = p 15
Hard-margin SVM: Dual problem 16 1 min w,w 0 2 w 2 s. t. y i w T x i + w 0 1 i = 1,, By incorporating the constraints through lagrangian multipliers, we will have: min w,w 0 max {α n 0} 1 2 w 2 + α n 1 y n (w T x (n) + w 0 ) Dual problem (changing the order of min and max in the above problem): max min 1 {α n 0} w,w 0 2 w 2 + α n 1 y n (w T x (n) + w 0 )
Hard-margin SVM: Dual problem max min {α n 0} w,w 0 L w, w 0, α L w, w 0, α = 1 2 w 2 + α n 1 y n (w T x (n) + w 0 ) w L w, w 0, α = 0 w α n y n x n = 0 w = α n y n x n L w,w 0,α = 0 w α n y (n) = 0 0 17 w 0 do not appear, instead, a global constraint on α is created.
Substituting w = α n y n x n α n y (n) = 0 In the Largrangian L w, w 0, α = 1 2 wt w + α n 1 y n (w T x (n) + w 0 ) 18
Substituting In the Largrangian L w, w 0, α = 1 2 wt w + w = α n y n x n α n y (n) = 0 α n 1 y n (w T x (n) + w 0 ) We get L w, w 0, α = α n 19
Substituting In the Largrangian L w, w 0, α = 1 2 wt w + w = α n y n x n α n y (n) = 0 α n 1 y n (w T x (n) + w 0 ) We get L w, w 0, α = α n 20
Substituting In the Largrangian L w, w 0, α = 1 2 wt w + w = α n y n x n α n y (n) = 0 α n 1 y n (w T x (n) + w 0 ) We get L α = α n 1 2 m=1 α n α m y (n) y (m) x n T x m Maximize w.r.t. α subject to α n 0 for n = 1,, and α n y (n) = 0 21
Hard-margin SVM: Dual problem max α α n 1 2 m=1 α n α m y (n) y (m) x n T x m Subject to α n y (n) = 0 α n 0 n = 1,, It is a convex QP 22
Solution Quadratic programming: 1 y 1 y 1 1 T x x 1 y 1 y x 1 T x min α 2 αt α + ( 1) T α y y 1 x T x 1 y y x T x s. t. α 0 y T α = 0 23
Finding the hyperplane After finding α by QP, we find w: How to find w 0? w = α n y n x n we discuss it after introducing support vectors 24
Karush-Kuhn-Tucker (KKT) conditions ecessary conditions for the solution [w, w 0, α ]: w L w, w 0, α w,w 0,α = 0 L w,w 0,α w 0 w,w 0,α = 0 α n 0 n = 1,, y n w T x n + w 0 1 n = 1,, α i 1 y n w T x n + w 0 = 0 n = 1,, 25 min f(x) x s. t. g i x 0 i = 1,, m L x, α = f x + α i g i x In general, the optimal x, α satisfies KKT conditions: x L x, α = 0 x,α α i 0 i = 1,, m g i x 0 i = 1,, m α i g i x = 0 i = 1,, m
Karush-Kuhn-Tucker (KKT) conditions Inactive constraint (α = 0) Active constraint 26 [wikipedia]
Hard-margin SVM: Support vectors Inactive constraint: y n w T x n + w 0 > 1 α n = 0 and thus x n is not a support vector. Active constraint: y n w T x n + w 0 = 1 α n can be greater than 0 and thus x i can be a support vector. x 2 α > 0 α > 0 α > 0 27 x 1
Hard-margin SVM: Support vectors Inactive constraint: y n w T x n + w 0 > 1 α n = 0 and thus x n is not a support vector. Active constraint: y n w T x n + w 0 = 1 x 2 α > 0 α = 0 α = 0 α > 0 α > 0 A sample with α n = 0 can also lie on one of the margin hyperplanes 28 x 1
Hard-margin SVM: Support vectors SupportVectors (SVs)= {x n α n > 0} The direction of hyper-plane can be found only based on support vectors: w = α n >0 α n y (n) x (n) x 2 29 x 1
Finding the hyperplane After finding α by QP, we find w: How to find w 0? w = α n y n x n Each of the samples that has α s > 0 is on the margin, thus we solve for w 0 using any of SVs: w T x s + w 0 = 1 y s w T x s + w 0 = 1 w 0 = y s w T x s 30
Hard-margin SVM: Dual problem Classifying new samples using only SVs Classification of a new sample x: y = sign w 0 + w T x y = sign w 0 + T α n y n x n x α n >0 y = sign(y (s) α n >0 α n y (n) x n T x (s) + w 0 α n >0 α n y n x n T x) Support vectors are sufficient to predict labels of new samples The classifier is based on the expansion in terms of dot products of x with support vectors. 31
Hard-margin SVM: Dual problem max α α n 1 2 m=1 α n α m y (n) y (m) x n T x m Subject to α n y (n) = 0 α n 0 n = 1,, Only the dot product of each pair of training data appears in the optimization problem An important property that is helpful to extend to non-linear SVM 32
In the transformed space max α α n 1 2 m=1 α n α m y n y m φ x n Subject to α n y (n) = 0 α n 0 n = 1,, T φ x m φ(. ) 33
Beyond linear separability When training samples are not linearly separable, it has no solution. How to extend it to find a solution even though the classes are not exactly linearly separable. 34
Beyond linear separability How to extend the hard-margin SVM to allow classification error Overlapping classes that can be approximately separated by a linear boundary oise in the linearly separable classes x 2 35 x 1 x 1
Beyond linear separability: Soft-margin SVM Minimizing the number of misclassified points?! P-complete Soft margin: Maximizing a margin while trying to minimize the distance between misclassified points and their correct margin plane 36
Error measure Margin violation amount ξ n (ξ n 0): y n w T x n + w 0 1 ξ n Total violation: ξ n 37
Soft-margin SVM: Optimization problem SVM with slack variables: allows samples to fall within the margin, but penalizes them 1 2 w 2 + C ξ n min w,w 0, ξ n s. t. y n w T x n + w 0 1 ξ n n = 1,, ξ n 0 x 2 ξ < 1 ξ n : slack variables 0 < ξ n < 1: if x n is correctly classified but inside margin ξ > 1 ξ n > 1: if x n is misclassifed 38 x 1
Soft-margin SVM linear penalty (hinge loss) for a sample if it is misclassified or lied in the margin tries to maintain ξ n small while maximizing the margin. always finds a solution (as opposed to hard-margin SVM) more robust to the outliers Soft margin problem is still a convex QP 39
Soft-margin SVM: Parameter C C is a tradeoff parameter: small C allows margin constraints to be easily ignored large margin large C makes constraints hard to ignore narrow margin C enforces all constraints: hard margin can be determined using a technique like crossvalidation C 40
Soft-margin SVM: Cost function 41 min w,w 0, ξ n 1 2 w 2 + C s. t. y n w T x n + w 0 1 ξ n n = 1,, ξ n 0 It is equivalent to the unconstrained optimization problem: 1 min w,w 0 2 w 2 + C ξ n max(0,1 y (n) (w T x (n) + w 0 ))
SVM loss function Hinge loss vs. 0-1 loss y = 1 max(0,1 y(w T x + w 0 )) 0-1 Loss Hinge Loss w T x + w 0 42
Lagrange formulation L w, w 0, ξ, α, β = 1 2 w 2 + C + ξ n α n 1 ξ n y n (w T x (n) + w 0 ) β n ξ n Minimize w.r.t. w, w 0, ξ and maximize w.r.t. α n 0 and β n 43 0 min w,w 0, ξ n 1 2 w 2 + C ξ n s. t. y n w T x n + w 0 1 ξ n n = 1,, ξ n 0
Lagrange formulation L w, w 0, ξ, α, β = 1 2 w 2 + C ξ n + α n 1 44
Soft-margin SVM: Dual problem max α α n 1 2 m=1 α n α m y (n) y (m) x n T x m Subject to α n y (n) = 0 0 α n C n = 1,, After solving the above quadratic problem, w is find as: w = α n y (n) x (n) 45
Soft-margin SVM: Support vectors SupportVectors: α n > 0 If 0 < α n < C (margin support vector) SVs on the margin y n w T x n + w 0 = 1 (ξ n = 0) If α = C (non-margin support vector) SVs on or over the margin y n w T x n + w 0 < 1 (ξ n > 0) C α n β n = 0 46
SVM: Summary Hard margin: maximizing margin Soft margin: handling noisy data and overlapping classes Slack variables in the problem Dual problems of hard-margin and soft-margin SVM Classifier decision in terms of support vectors Dual problems lead us to non-linear SVM method easily by kernel substitution 47
ot linearly separable data oisy data or overlapping classes (we discussed about it: soft margin) ear linearly separable x 2 x 1 on-linear decision surface x 2 Transform to a new feature space 48 x 1
onlinear SVM Assume a transformation φ: R d R m space x φ x on the feature Find a hyper-plane in the transformed feature space: x 2 φ x = [φ 1 (x),..., φ m (x)] {φ 1 (x),...,φ m (x)}: set of basis functions (or features) φ i x : R d R φ 2 (x) φ: x φ x w T φ x + w 0 = 0 49 x 1 φ 1 (x)
Soft-margin SVM in a transformed space: Primal problem Primal problem: 1 min w,w 0 2 w 2 + C s. t. y n w T φ(x n ) + w 0 1 ξ n n = 1,, ξ n 0 ξ n w R m : the weights that must be found If m d (very high dimensional feature space) then there are many more parameters to learn 50
Soft-margin SVM in a transformed space: Dual problem Optimization problem: max α α n 1 2 m=1 Subject to α n y (n) = 0 α n α m y (n) y (m) φ x (n) T φ x (m) 0 α n C n = 1,, If we have inner products φ x (i) T φ x (j), only α = [α 1,, α ] needs to be learnt. not necessary to learn m parameters as opposed to the primal problem 51
Classifying a new data y = sign w 0 + w T φ(x) where w = αn >0 α n y (n) φ(x (n) ) and w 0 = y (s) w T φ(x (s) ) 52
Kernel SVM Learns linear decision boundary in a high dimension space without explicitly working on the mapped data Let φ x T φ x = K(x, x ) (kernel) Example: x = x 1, x 2 and second-order φ: φ x = 1, x 1, x 2, x 1 2, x 2 2, x 1 x 2 K x, x = 1 + x 1 x 1 + x 2 x 2 + x 1 2 x 1 2 + x 2 2 x 2 2 + x 1 x 1 x 2 x 2 53
Kernel trick Compute K x, x without transforming x and x Example: Consider K x, x = 1 + x T x 2 = 1 + x 1 x 1 + x 2 x 2 2 = 1 + 2x 1 x 1 + 2x 2 x 2 + x 1 2 x 1 2 + x 2 2 x 2 2 + 2x 1 x 1 x 2 x 2 This is an inner product in: φ x = 1, 2x 1, 2x 2, x 1 2, x 2 2, 2x 1 x 2 φ x = 1, 2x 1, 2x 2, x 1 2, x 2 2, 2x 1 x 2 54
Polynomial kernel: Degree two We instead use K(x, x ) = x T x + 1 2 that corresponds to: d-dimensional feature space x = x 1,,x d T φ x = 1, 2x 1,, 2x d, x 2 1,.., x 2 T d, 2x 1 x 2,, 2x 1 x d, 2x 2 x 3,, 2x d 1 x d 55
Polynomial kernel This can similarly be generalized to d-dimensioan x and φs are polynomials of order M: K x, x = 1 + x T x M = 1 + x 1 x 1 + x 2 x 2 + + x d x d M Example: SVM boundary for a polynomial kernel w 0 + w T φ x = 0 w 0 + αi >0 α i y (i) φ x i T φ x = 0 w 0 + αi >0 α i y (i) k(x i, x) = 0 w 0 + αi >0 α i y (i) 1 + x (i)t x M = 0 Boundary is a polynomial of order M 56
Why kernel? kernel functions K can indeed be efficiently computed, with a cost proportional to d (the dimensionality of the input) instead of m. Example: consider the second-order polynomial transform: φ x = 1, x 1,, x d, x 1 2, x 1 x 2,, x d x d T m = 1 + d + d 2 φ x T φ x = 1 + d i=1 x i x i + d i=1 d d j=1 x i x j x i x j d x i x i x j x j O(m) i=1 j=1 φ x T φ x = 1 + x T x + x T x 2 O(d) 57
Gaussian or RBF kernel If K x, x is an inner product in some transformed space of x, it is good K x, x = exp( x x 2 ) Take one dimensional case with γ = 1: K x, x = exp x x 2 γ = exp x 2 exp x 2 exp 2xx = exp x 2 exp x 2 2 k x k x k k! k=1 58
Some common kernel functions Linear: k(x, x ) = x T x Polynomial: k x, x = (x T x + 1) M Gaussian: k x, x = exp( x x 2 ) γ Sigmoid: k x, x = tanh(ax T x + b) 59
Kernel formulation of SVM Optimization problem: max α α n 1 2 m=1 Subject to α n y (n) = 0 k(x n, x (m) ) α n α m y (n) y (m) φ x (n) T φ x (m) 0 α n C n = 1,, Q = y 1 y 1 K x 1, x 1 y 1 y K x, x 1 y y 1 K x, x 1 y y K x, x 60
Classifying a new data y = sign w 0 + w T φ(x) where w = αn >0 α n y (n) φ(x (n) ) and w 0 = y (s) w T φ(x (s) ) y = sign w 0 + w 0 = y (s) α n >0 α n >0 α n y n φ x n α n y n φ x n T φ(x) k(x n, x) T φ x s k(x n, x (s) ) 61
Gaussian kernel Example: SVM boundary for a gaussian kernel Considers a Gaussian function around each data point. w 0 + αi >0 α i y (i) exp( x x(i) 2 ) = 0 σ SVM + Gaussian Kernel can classify any arbitrary training set Training error is zero when σ 0 All samples become support vectors (likely overfiting) 62
Hard margin Example For narrow Gaussian (large σ), even the protection of a large margin cannot suppress overfitting. 63
SVM Gaussian kernel: Example f x = w 0 + α i >0 α i y (i) exp( x x(i) 2 2σ 2 ) 64 This example has been adopted from Zisserman s slides
SVM Gaussian kernel: Example 65 This example has been adopted from Zisserman s slides
SVM Gaussian kernel: Example 66 This example has been adopted from Zisserman s slides
SVM Gaussian kernel: Example 67 This example has been adopted from Zisserman s slides
SVM Gaussian kernel: Example 68 This example has been adopted from Zisserman s slides
SVM Gaussian kernel: Example 69 This example has been adopted from Zisserman s slides
SVM Gaussian kernel: Example 70 This example has been adopted from Zisserman s slides
Kernel trick: Idea Kernel trick Extension of many well-known algorithms to kernel-based ones By substituting the dot product with the kernel function k x, x = φ x T φ(x ) k x, x shows the dot product of x and x in the transformed space. Idea: when the input vectors appears only in the form of dot products, we can use kernel trick Solving the problem without explicitly mapping the data Explicit mapping is expensive if φ x is very high dimensional 71
Kernel trick: Idea (Cont d) Instead of using a mapping φ: X F to represent x X by φ(x) F, a similarity function k: X X R is used. We specify only an inner product function between points in the transformed space (not their coordinates) In many cases, the inner product in the embedding space can be computed efficiently. 72
Constructing kernels Construct kernel functions directly Ensure that it is a valid kernel Corresponds to an inner product in some feature space. Example: k(x, x ) = x T x 2 Corresponding mapping: φ x = x 2 1, 2 2x 1 x 2, x T 2 for x = x 1, x T 2 We need a way to test whether a kernel is valid without having to construct φ x 73
Valid kernel: ecessary & sufficient conditions Gram matrix K : K ij = k(x (i), x (j) ) [Shawe-Taylor & Cristianini 2004] Restricting the kernel function to a set of points {x 1, x 2,, x () } K = k(x (1), x (1) ) k(x (1), x () ) k(x (), x (1) ) k(x (), x () ) Mercer Theorem: The kernel matrix is Symmetric Positive Semi-Definite (for any choice of data points) Any symmetric positive definite matrix can be regarded as a kernel matrix, that is as an inner product matrix in some space 74
Extending linear methods to kernelized ones Kernelized version of linear methods Linear methods are famous Unique optimal solutions, faster learning algorithms, and better analysis However, we often require nonlinear methods in real-world problems and so we can use kernel-based version of these linear algorithms Replacing inner products with kernels in linear algorithms very flexible methods We can operate in the mapped space without ever computing the coordinates of the data in that space 75
Example: kernelized minimum distance classifier If x μ 1 < x μ 2 then assign x to C 1 x μ 1 T x μ 1 < x μ 2 T x μ 2 2x T μ 1 + μ 1 T μ 1 < 2x T μ 2 + μ 2 T μ 2 2 n + y n =1 y m x =1 < 2 y n =2 xt x n + y n =2 y m =2 1 1 1 2 2 2 y n =1 xt x n T x m x n T x m 2 y n K x, x n =1 + y n =1 y m =1 K x n, x m < 2 y n K x, x n =2 + y n =2 y m =2 K x n, x m 1 1 1 2 2 2 76
Which information can be obtained from kernel? Example: we know all pairwise distances d φ x, φ z 2 = φ x φ z 2 = k x, x + k z, z 2k(x, z) Therefore, we also know distance of points from center of mass of a set Many dimensionality reduction, clustering, and classification methods can be described according to pairwise distances. This allow us to introduce kernelized versions of them 77
Example: Kernel ridge regression min w w T x n y n 2 + λw T w 2x n w T x n y n + 2λw w = α n x n α n = 1 λ wt x n y n 78
Example: Kernel ridge regression (Cont d) min w Dual representation: w T φ x n y n 2 + λw T w J α = α T ΦΦ T ΦΦ T α 2α T ΦΦ T y + y T y + λα T ΦΦ T α J α = α T KKα 2α T Ky + y T y + λα T Kα α J α = 0 α = K + λi 1 y w = α n φ x n 79
Example: Kernel ridge regression (Cont d) Prediction for new x: f x = w T φ x w = Φ T α = K(x (1), x) K(x (), x) = α T Φφ x T K + λi 1 y 80
Kernels for structured data Kernels also can be defined on general types of data Kernel functions do not need to be defined over vectors just we need a symmetric positive definite matrix Thus, many algorithms can work with general (non-vectorial) data Kernels exist to embed strings, trees, graphs, This may be more important than nonlinearity kernel-based version of classical learning algorithms for recognition of structured data 81
Kernel function for objects Sets: Example of kernel function for sets: k A, B = 2 A B Strings: The inner product of the feature vectors for two strings can be defined as e.g. sum over all common subsequences weighted according to their frequency of occurrence and lengths A E G A T E A G G E G T E A G A E G A T G 82
Kernel trick advantages: summary Operating in the mapped space without ever computing the coordinates of the data in that space Besides vectors, we can introduce kernel functions for structured data (graphs, strings, etc.) Much of the geometry of the data in the embedding space is contained in all pairwise dot products In many cases, inner product in the embedding space can be computed efficiently. 83
Resources C. Bishop, Pattern Recognition and Machine Learning, Chapter 6.1-6.2, 7.1. Yaser S. Abu-Mostafa, et al., Learning from Data, Chapter 8. 84