Linear Discriminant Functions and Support Vector Machines Linear, threshold units CSE19, Winter 11 Biometrics CSE 19 Lecture 11 1 X i : inputs W i : weights θ : threshold 3 4 5 1
6 7 Courtesy of University of Teas at Austin 8 -layer Neural Net Reading: Duda, Hart & Stork, Sec. 5.11 "A Tutorial on Support Vector Machines for Pattern Recognition," Christopher Burges 9 Perceptron Revisited: Linear Separators Binary classification can be viewed as the task of separating classes in feature space: Linear Separators Which of the linear separators is optimal? w T + b > w T + b = w T + b < f() = sign(w T + b) 1 11
Classification Margin Distance from eample i to the separator is r = wt i + b w Training eamples closest to the hyperplane are support vectors. Margin ρ of the separator is the distance from the separator to support vectors. ρ Maimum Margin Classification Maimizing the margin is good according to intuition and PAC (Probably Approimately Correct) theory. Implies that only support vectors matter; other training eamples are ignorable. r 1 Solved as a Quadratic Programming Problem 13 Linear SVM Mathematically Let training set {( i, y i )} i=1..n, i R d, y i {-1, 1} be separated by a hyperplane with margin ρ. Then for each training eample ( i, y i ): w T i + b - ρ/ if y i = -1 w T i + b ρ/ if y i = 1 y i (w T i + b) ρ/ For every support vector s the above inequality is an equality. After rescaling w and b by ρ/ in the equality, we obtain that distance between each s and the hyperplane is Then the margin can be epressed through (rescaled) w and b as: Linear SVMs Mathematically (cont.) Then we can formulate the quadratic optimization problem: is maimized and for all ( i, y i ), i=1..n : y i (w T i + b) 1 Which can be reformulated as: Φ(w) = w =w T w is minimized and for all ( i, y i ), i=1..n : y i (w T i + b) 1 14 15 Solving the Optimization Problem Φ(w) =w T w is minimized and for all ( i, y i ), i=1..n : y i (w T i + b) 1 Need to optimize a quadratic function subject to linear constraints. Quadratic optimization problems are a well-known class of mathematical programming problems for which several (non-trivial) algorithms eist. The solution involves constructing a dual problem where a Lagrange multiplier α i is associated with every inequality constraint in the primal (original) problem: Find α 1 α n such that Q(α) =Σα i - ½ΣΣα i α j y i y j it j is maimized and () α i for all α i 16 The Optimization Problem Solution Given a solution α 1 α n to the dual problem, solution to the primal is: w =Σα i y i i b = y k - Σα i y i i T k for any α k > Each non-zero α i indicates that corresponding i is a support vector. Then the classifying function is (note that we don t need w eplicitly): f() = Σα i y i it + b Notice that it relies on an inner product between the test point and the support vectors i we will return to this later. Also keep in mind that solving the optimization problem involved computing the inner products it j between all training points. 17 3
Soft Margin Classification What if the training set is not linearly separable? Slack variables ξ i can be added to allow misclassification of difficult or noisy eamples, resulting margin called soft. Soft Margin Classification Mathematically The old formulation: Φ(w) =w T w is minimized and for all ( i,y i ), i=1..n : y i (w T i + b) 1 ξ i ξ i Modified formulation incorporates slack variables: Φ(w) =w T w + CΣξ i is minimized and for all ( i,y i ), i=1..n : y i (w T i + b) 1 ξ i,, ξ i Parameter C can be viewed as a way to control overfitting: it trades off the relative importance of maimizing the margin and fitting the training data. 18 19 Soft Margin Classification Solution Dual problem is identical to separable case (would not be identical if the - norm penalty for slack variables CΣξ i was used in primal objective, we would need additional Lagrange multipliers for slack variables): Find α 1 α N such that Q(α) =Σα i - ½ΣΣα i α j y i y j it j is maimized and () α i C for all α i Again, i with non-zero α i will be support vectors. Solution to the dual problem is: w =Σα i y i i b= y k (1- ξ k ) - Σα i y i it k for any k s.t. α k > Again, we don t need to compute w eplicitly for classification: f() = Σα i y i it + b Linear SVMs: Overview The classifier is a separating hyperplane. Most important training points are support vectors; they define the hyperplane. Quadratic optimization algorithms can identify which training points i are support vectors with non-zero Lagrangian multipliers α i. Both in the dual formulation of the problem and in the solution training points appear only inside inner products: Find α 1 α N such that Q(α) =Σα i - ½ΣΣα i α j y i y j it j is maimized and () α i C for all α i f() = Σα i y i it + b 1 Non-linear SVMs Datasets that are linearly separable with some noise work out great: But what are we going to do if the dataset is just too hard? Non-linear SVMs: Feature spaces General idea: the original feature space can always be mapped to some higher-dimensional feature space where the training set is separable: Φ: φ() How about mapping data to a higher-dimensional space: Video Eample: http://www.youtube.com/watch?v=3licbrzprza 3 4
The Kernel Trick The linear classifier relies on inner product between vectors K( i, j )= it j If every datapoint is mapped into high-dimensional space via some transformation Φ: φ(), the inner product becomes: K( i, j )= φ( i ) T φ( j ) A kernel function is a function that is eqiuvalent to an inner product in some feature space. Eample: -dimensional vectors =[ 1 ]; let K( i, j )=(1 + it j ), Need to show that K( i, j )= φ( i ) T φ( j ): K( i, j )=(1 + it j ),= 1+ i1 j1 + i1 j1 i j + i j + i1 j1 + i j = = [1 i1 i1 i i i1 i ] T [1 j1 j1 j j j1 j ] = = φ( i ) T φ( j ), where φ() = [1 1 1 1 ] Thus, a kernel function implicitly maps data to a high-dimensional space (without the need to compute each φ() eplicitly). 4 What Functions are Kernels? For some functions K( i, j ) checking that K( i, j )= φ( i ) T φ( j ) can be cumbersome. Mercer s theorem: Every semi-positive definite symmetric function is a kernel Semi-positive definite symmetric functions correspond to a semi-positive definite symmetric Gram matri: K= K( 1, 1 ) K( 1, ) K( 1, 3 ) K( 1, n ) K(, 1 ) K(, ) K(, 3 ) K(, n ) K( n, 1 ) K( n, ) K( n, 3 ) K( n, n ) 5 Eamples of Kernel Functions Linear: K( i, j )= it j Mapping Φ: φ(), where φ() is itself Polynomial of power p: K( i, j )= (1+ it j ) p Mapping Φ: φ(), where φ() has dimensions Dual problem formulation: Non-linear SVMs Mathematically Find α 1 α n such that Q(α) =Σα i - ½ΣΣα i α j y i y j K( i, j ) is maimized and () α i for all α i Gaussian (radial-basis function): K( i, j ) = Mapping Φ: φ(), where φ() is infinite-dimensional: every point is mapped to a function (a Gaussian); combination of functions for support vectors is the separator. Higher-dimensional space still has intrinsic dimensionality d (the mapping is not onto), but linear separators in it correspond to non-linear separators in original space. 6 The solution is: f() = Σα i y i K( i, j )+ b Optimization techniques for finding α i s remain the same! 7 SVM applications SVMs were originally proposed by Boser, Guyon and Vapnik in 199 and gained increasing popularity in late 199s. SVMs are currently among the best performers for a number of classification tasks ranging from tet to genomic data. SVMs can be applied to comple data types beyond feature vectors (e.g. graphs, sequences, relational data) by designing kernel functions for such data. SVM techniques have been etended to a number of tasks such as regression [Vapnik et al. 97], principal component analysis [Schölkopf et al. 99], etc. Tuning SVMs remains a black art: selecting a specific kernel and parameters is usually done in a try-and-see manner. Implementations: See libsvm, svmlight, and others 8 5