Support Vector Machine (SVM) and Kernel Methods

Support Vector Machine (SVM) and Kernel Methods CE-717: Machine Learning Sharif University of Technology Fall 2014 Soleymani

Outline Margin concept Hard-Margin SVM Soft-Margin SVM Dual Problems of Hard-Margin SVM and Soft-Margin SVM Nonlinear SVM Kernel trick Kernel methods 2

Margin Which line is better to select as the boundary to provide more generalization capability? Larger margin provides better generalization to unseen data x 2 Margin for a hyperplane that separates samples of two linearly separable classes is: The smallest distance between the decision boundary and any of the training samples x 1 3

Maximum margin SVM finds the solution with maximum margin Solution: a hyperplane that is farthest from all training samples x 2 x 2 x 1 Larger margin x 1 The hyperplane with the largest margin has equal distances to the nearest sample of both classes 4

Hard-margin SVM: Optimization problem 2M max M,w,w 0 w s. t. w T x i + w 0 M x i C 1 y i = 1 w T x i + w 0 M x i C 2 y i = 1 w T x + w 0 = 0 x 2 M w Margin: 2 M w 5 w w T x + w 0 = M w T x + w 0 = M x 1

Hard-margin SVM: Optimization problem 2M max M,w,w 0 w s. t. y (i) w T x i + w 0 M i = 1,, N w T x + w 0 = 0 x 2 M w 6 w w T x + w 0 = M w T x + w 0 = M x 1

Hard-margin SVM: Optimization problem We can set w = w M, w 0 = w 0 M : max w,w 0 2 w s. t. y (i) w T x i + w 0 1 i = 1,, N w T x + w 0 = 0 x 2 1 w The place of boundary and margin lines do not change w w T x + w 0 = 1 w T x + w 0 = 1 7 x 1

Hard-margin SVM: Optimization problem We can equivalently optimize: 1 min w,w 0 2 w 2 s. t. y i w T x i + w 0 1 i = 1,, N It is a convex Quadratic Programming (QP) problem There are computationally efficient packages to solve it. It has a global minimum (if any). When training samples are not linearly separable, it has no solution. How to extend it to find a solution even though the classes are not linearly separable. 8

Beyond linear separability How to extend the hard-margin SVM to allow classification error Overlapping classes that can be approximately separated by a linear boundary Noise in the linearly separable classes x 2 9 x 1 x 1

Beyond linear separability: Soft-margin SVM Minimizing the number of misclassified points?! NP-complete Soft margin: Maximizing a margin while trying to minimize the distance between misclassified points and their correct margin plane 10

Soft-margin SVM SVM with slack variables: allows samples to fall within the margin, but penalizes them N 1 2 w 2 + C ξ i min w,w 0, ξ N i i=1 i=1 s. t. y i w T x i + w 0 1 ξ i i = 1,, N ξ i 0 x 2 ξ i : slack variables ξ i > 1: if x i misclassifed ξ i 0 < ξ i < 1 : if x i correctly classified but inside margin 11 x 1

Soft-margin SVM linear penalty (hinge loss) for a sample if it is misclassified or lied in the margin tries to maintain ξ i small while maximizing the margin. always finds a solution (as opposed to hard-margin SVM) more robust to the outliers Soft margin problem is still a convex QP ξ = 0 ξ = 0 12

Soft-margin SVM: Parameter C C is a tradeoff parameter: small C allows margin constraints to be easily ignored large margin large C makes constraints hard to ignore narrow margin C = enforces all constraints: hard margin can be determined using a technique like crossvalidation C 13

Soft-margin SVM: Cost function min w,w 0, ξ N i i=1 1 2 w 2 + C i=1 s. t. y i w T x i + w 0 1 ξ i i = 1,, n ξ i 0 It is equivalent to the unconstrained optimization problem: n ξ i 1 min w,w 0 2 w 2 + C n i=1 max(0,1 y (i) (w T x (i) + w 0 )) 14

SVM loss function Hinge loss vs. 0-1 loss y = 1 max(0,1 y(w T x + w 0 )) 0-1 Loss Hinge Loss w T x + w 0 15

Optimization: Lagrangian multipliers p = min f(x) x s. t. g i x 0 i = 1,, m h i x = 0 i = 1,, p L x, α, λ = f x + m i=1 α i g i x + p i=1 Lagrangian multipliers λ i h i x max L x, α, λ = α i 0, λ i any g i x > 0 any h i x 0 f x otherwise p = min x max α i 0, λ i L x, α, λ α = α 1,, α m λ = [λ 1,, λ p ] 16

Optimization: Dual problem In general, we have: max min x y f(x, y) min y max x f(x, y) Primal problem: p = min x max α i 0, λ i L x, α, λ Dual problem: d = max α i 0, λ i min x L x, α, λ Obtained by swapping the order of the optimizations d p When the original problem is convex (f and g are convex functions and h is affine), we have strong duality d = p 17

Hard-margin SVM: Dual problem 18 1 min w,w 0 2 w 2 s. t. y i w T x i + w 0 1 i = 1,, N By incorporating the constraints through lagrangian multipliers, we will have: min w,w 0 max {α i 0} 1 2 w 2 + N i=1 α i 1 y i (w T x (i) + w 0 ) Dual problem (changing the order of min and max in the above problem): max min 1 {α i 0} w,w 0 2 w 2 + N i=1 α i 1 y i (w T x (i) + w 0 )

Hard-margin SVM: Dual problem max min {α i 0} w,w 0 J w, w 0, α J w, w 0, α = 1 2 w 2 + N i=1 α i 1 y i (w T x (i) + w 0 ) w J w, w 0, α = 0 w = N i=1 α i y (i) x (i) J w,w 0,α = 0 N w i=1 α i y (i) = 0 0 19 w 0 do not appear, instead, a global constraint on α is created.

Hard-margin SVM: Dual problem max α N i=1 N α i 1 2 i=1 N j=1 Subject to N i=1 α i y (i) = 0 α i α j y (i) y (j) x i T x j α i 0 i = 1,, N It is a convex QP By solving the above problem first we find α and then w N α i y (i) x (i) = i=1 w 0 can be set by making the margin equidistant to classes. w 0 is found using each of equations on SVs: y i w T x i + w 0 = 1 Numerically safer to find w 0 using the equations on all SVs 20

Hard-margin SVM: Support vectors SupportVectors (SVs)= {x i α i > 0} The direction of hyper-plane can be found only based on support vectors: w = α i >0 α i y (i) x (i) 21

Hard-margin SVM: Dual problem Classifying new samples using only SVs Classification of a new sample x: y = sign w 0 + w T x y = sign w 0 + y = sign(w 0 + T α i y i x i x α i >0 α i >0 α i y i x i T x) Support vectors are sufficient to predict labels of new samples The classifier is based on the expansion in terms of dot products of x with support vectors. 22

Karush-Kuhn-Tucker (KKT) conditions Necessary conditions for the solution [w, w 0, α ]: w L w, w 0, α = 0 L w,w 0,α w 0 = 0 α i 0 i = 1,, N y i w T x i + w 0 1 i = 1,, N α i 1 y i w T x i + w 0 = 0 i = 1,, N 23

Hard-margin SVM: Support vectors Inactive constraint: y i w T x i + w 0 > 1 α i = 0 and thus x i is not a support vector. Active constraint: y i w T x i + w 0 = 1 α i can be greater than 0 and thus x i can be a support vector. x 2 α > 0 α > 0 α > 0 1 w 24 x 1

Hard-margin SVM: Dual problem N N N max α i=1 α i 1 2 i=1 j=1 α i α j y (i) y (j) x i T x j Subject to N i=1 α i y (i) = 0 α i 0 i = 1,, N Only the dot product of each pair of training data appears in the optimization problem This is an important property that is helpful to extend to non-linear SVM (the cost function does not depend explicitly on the dimensionality of the feature space). 26

Soft-margin SVM: Dual problem max α N i=1 N α i 1 2 i=1 N j=1 α i α j y (i) y (j) x i T x j Subject to N i=1 α i y (i) = 0 0 α i C i = 1,, N By solving the above quadratic problem first we find α and then find w = N i=1 α i y (i) x (i) and w 0 is computed from SVs. For a test sample x (as before): y = sign w 0 + w T x = sign(w 0 + α i >0 α i y i x i T x) 27

Soft-margin SVM: Support vectors SupportVectors: α > 0 If 0 < α < C: SVs on the margin, ξ = 0. If α = C: SVs on or over the margin. 28

Primal vs. dual hard-margin SVM problem Primal problem of hard-margin SVM N inequality constraints d + 1 number of variables Dual problem of hard-margin SVM one equality constraint N positivity constraints N number of variables (Lagrange multipliers) Objective function more complicated The dual problem is helpful and instrumental to use the kernel trick 29

Primal vs. dual soft-margin SVM problem Primal problem of soft-margin SVM N inequality constraints N positivity constraints N + d + 1 number of variables Dual problem of soft-margin SVM one equality constraint 2N positivity constraints N number of variables (Lagrange multipliers) Objective function more complicated The dual problem is helpful and instrumental to use the kernel trick 30

Not linearly separable data Noisy data or overlapping classes (we discussed about it: soft margin) Near linearly separable x 2 x 1 Non-linear decision surface x 2 Transform to a new feature space 31 x 1

Nonlinear SVM Assume a transformation φ: R d R m space x φ x on the feature Find a hyper-plane in the transformed feature space: x 2 φ x = [φ 1 (x),..., φ m (x)] {φ 1 (x),...,φ m (x)}: set of basis functions (or features) φ i x : R d R φ 2 (x) φ: x φ x w T φ x + w 0 = 0 32 x 1 φ 1 (x)

Soft-margin SVM in a transformed space: Primal problem Primal problem: 1 min w,w 0 2 w 2 + C i=1 s. t. y i w T φ(x i ) + w 0 1 ξ i i = 1,, N ξ i 0 N ξ i w R m : the weights that must be found If m d (very high dimensional feature space) then there are many more parameters to learn 33

Soft-margin SVM in a transformed space: Dual problem Optimization problem: max α N i=1 N α i 1 2 i=1 N j=1 α i α j y (i) y (j) φ x (i) T φ x (i) Subject to N i=1 α i y (i) = 0 0 α i C i = 1,, n If we have inner products φ x (i) T φ x (j), only α = [α 1,, α N ] needs to be learnt. 34 It is not necessary to learn m parameters as opposed to the primal problem

Kernelized soft-margin SVM Optimization problem: max α N i=1 N α i 1 2 i=1 N j=1 k x, x = φ x T φ(x ) α i α j y (i) y (j) φ k(x x (i) i T, φx (j) x (i) ) Subject to N i=1 α i y (i) = 0 0 α i C i = 1,, n Classifying a new data: y = sign w 0 + w T φ(x) = sign(w 0 + αi >0 α i y i φ(x i ) T φ(x)) 35 k(x i, x)

Kernel trick Kernel: the dot product in the mapped feature space F k x, x = φ x T φ(x ) φ x = φ 1 x,, φ m x T k x, x shows the similarity of x and x. The distance can be found as accordingly Kernel trick Extension of many well-known algorithms to kernel-based ones By substituting the dot product with the kernel function 36

Kernel trick: Idea Idea: when the input vectors appears only in the form of dot products, we can use other choices of kernel function instead of inner product Solving the problem without explicitly mapping the data Explicit mapping is expensive if φ x is very high dimensional Instead of using a mapping φ: X F to represent x X by φ(x) F, a similarity function k: X X R is used. The data set can be represented by N N matrix K. We specify only an inner product function between points in the transformed space (not their coordinates) In many cases, the inner product in the embedding space can be computed efficiently. 37

Some common kernel functions Linear: k(x, x ) = x T x Polynomial: k x, x = (x T x + c) M c 0 Gaussian: k x, x = exp( x x 2 2σ 2 ) Sigmoid: k x, x = tanh(ax T x + b) 38

Polynomial kernel: Degree two x = x 1 x 2 φ x = x 1 2 x 2 2 2x 1 x 2 φ x T φ z = x 1 2 z 1 2 + x 2 2 z 2 2 + 2x 1 x 2 z 1 z 2 k x, z = x T z 2 = x 1 2 z 1 2 + x 2 2 z 2 2 + 2x 1 x 2 z 1 z 2 The dot product in this transformed space is equal to the squared dot product in the input space. k x, z = x T z 2 contains only terms of degree two 39

Polynomial kernel: Degree two We instead use k(x, z) = x T z + 1 2 that corresponds to: x 1 2 x 2 2 x = x 1 x 2 φ x = 2x 1 x 2 2x 1 2x 2 1 In general, k(x, z) = x T z + c 2 (c > 0) contains constant and linear terms as well as all terms of order two. In a d-dimensional feature space, x = x 1,, x d T : φ x = x 1 2,.., x d 2, 2x 1 x 2,, 2x 1 x d, 2x 2 x 3,, 2x d 1 x d, 2x 1,, 2x d, 1 T 40

Polynomial kernel In general, k(x, z) = x T z M contains all monomials of order M. This can similarly be generalized to include all terms up to degree M, k(x, z) = x T z + c M (c > 0) Example: SVM boundary for a polynomial kernel w 0 + w T φ x = 0 w 0 + αi >0 α i y (i) φ x i T φ x = 0 w 0 + αi >0 α i y (i) k(x i, x) = 0 w 0 + αi >0 α i y (i) x (i)t x + c M = 0 Boundary is a polynomial of order M 41

Gaussian kernel Gaussian: k x, x = exp( x x 2 2σ 2 ) Infinite dimensional feature space Example: SVM boundary for a gaussian kernel Considers a Gaussian function around each data point. w 0 + αi >0 α i y (i) exp( x x(i) 2 ) = 0 2σ 2 SVM + Gaussian Kernel can classify any arbitrary training set Training error is zero when σ 0 All samples become support vectors (likely overfiting) 42

SVM: Linear and Gaussian kernels example Linear kernel: C = 0.1 Gaussian kernel: C = 1,σ = 0.25 some SVs appear to be far from the boundary? 43 [Zisserman]

SVM Gaussian kernel: Example f x = w 0 + α i >0 α i y (i) exp( x x(i) 2 2σ 2 ) 44 Source: Zisserman s slides

SVM Gaussian kernel: Example 45 Source: Zisserman s slides

SVM Gaussian kernel: Example 46 Source: Zisserman s

SVM Gaussian kernel: Example 47 Source: Zisserman s slides

SVM Gaussian kernel: Example 48 Source: Zisserman s slides

SVM Gaussian kernel: Example 49 Source: Zisserman s slides

SVM Gaussian kernel: Example 50 Source: Zisserman s slides

Constructing kernels Construct kernel functions directly Ensure that it is a valid kernel Corresponds to an inner product in some feature space. Example: k(x, x ) = x T x 2 Corresponding mapping: φ x = x 2 1, 2 2x 1 x 2, x T 2 for x = x 1, x T 2 We need a way to test whether a kernel is valid without having to construct φ x 51

Valid kernel: Necessary & sufficient conditions Gram matrix K N N : K ij = k(x (i), x (j) ) [Shawe-Taylor & Cristianini 2004] Restricting the kernel function to a set of points {x 1, x 2,, x (N) } K = k(x (1), x (1) ) k(x (1), x (N) ) k(x (N), x (1) ) k(x (N), x (N) ) Mercer Theorem: The kernel matrix is Symmetric Positive Semi-Definite (for any choice of data points) Any symmetric positive definite matrix can be regarded as a kernel matrix, that is as an inner product matrix in some space 52

Construct Valid Kernels c > 0, k 1: valid kernel f(. ): any function q(. ): a polynomial with coefficients 0 k 1, k 2: valid kernels φ(x): a function from x to R M k3(.,. ): a valid kernel in R M A : a symmetric positive semi-definite matrix x a and x b are variables (not necessarily disjoint) with x = (x a, x b ), and k a and k b are valid kernel functions over their respective spaces. 53 [Bishop]

Extending linear methods to kernelized ones Kernelized version of linear methods Linear methods are famous Unique optimal solutions, faster learning algorithms, and better analysis However, we often require nonlinear methods in real-world problems and so we can use kernel-based version of these linear algorithms If a problem can be formulated such that feature vectors only appear in inner products, we can extend it to a kernelized one When we only need to know the dot product of all pairs of data (about the geometry of the data in the feature space) By replacing inner products with kernels in linear algorithms, we obtain very flexible methods We can operate in the mapped space without ever computing the coordinates of the data in that space 54

Which information can be obtained from kernel? Example: we know all pairwise distances d φ x, φ z 2 = φ x φ z 2 = k x, x + k z, z 2k(x, z) Therefore, we also know distance of points from center of mass of a set Many dimensionality reduction, clustering, and classification methods can be described according to pairwise distances. This allow us to introduce kernelized versions of them 55

Kernels for structured data Kernels also can be defined on general types of data Kernel functions do not need to be defined over vectors just we need a symmetric positive definite matrix Thus, many algorithms can work with general (non-vectorial) data Kernels exist to embed strings, trees, graphs, This may be more important than nonlinearity that we can use kernel-based version of the classical learning algorithms for recognition of structured data 56

Kernel function for objects Sets: Example of kernel function for sets: k A, B = 2 A B Strings: The inner product of the feature vectors for two strings can be defined as e.g. sum over all common subsequences weighted according to their frequency of occurrence and lengths A E G A T E A G G E G T E A G A E G A T G 57

Kernel trick advantages: summary Operating in the mapped space without ever computing the coordinates of the data in that space Besides vectors, we can introduce kernel functions for structured data (graphs, strings, etc.) Much of the geometry of the data in the embedding space is contained in all pairwise dot products In many cases, inner product in the embedding space can be computed efficiently. 58

SVM: Summary Hard margin: maximizing margin Soft margin: handling noisy data and overlapping classes Slack variables in the problem Dual problems of hard-margin and soft-margin SVM Lead us to non-linear SVM method easily by kernel substitution Also, classifier decision in terms of support vectors Kernel SVM s Learns linear decision boundary in a high dimension space without explicitly working on the mapped data 59