Max Margin-Classifier

Transcription

1 Max Margin-Classifier Oliver Schulte - CMPT 726 Bishop PRML Ch. 7

2 Outline Maximum Margin Criterion Math Maximizing the Margin Non-Separable Data

3 Kernels and Non-linear Mappings Where does the maximization problem come from? The intuition comes from the primal version, which is based on a feature mapping φ. Theorem: Every valid kernel k(x, y) is the dot-product φ(x)[φ(x)] T for some set of basis functions (feature mapping) φ. The feature space φ(x) could be high-dimensional, even infinite. This is good because if data aren t separable in original input space (x), they may be in feature space φ(x) We can think about how to find a good linear separator using the dot product in high dimensions, then transfer this back to kernels in the original input space.

4 Kernels and Non-linear Mappings Where does the maximization problem come from? The intuition comes from the primal version, which is based on a feature mapping φ. Theorem: Every valid kernel k(x, y) is the dot-product φ(x)[φ(x)] T for some set of basis functions (feature mapping) φ. The feature space φ(x) could be high-dimensional, even infinite. This is good because if data aren t separable in original input space (x), they may be in feature space φ(x) We can think about how to find a good linear separator using the dot product in high dimensions, then transfer this back to kernels in the original input space.

5 Kernels and Non-linear Mappings Where does the maximization problem come from? The intuition comes from the primal version, which is based on a feature mapping φ. Theorem: Every valid kernel k(x, y) is the dot-product φ(x)[φ(x)] T for some set of basis functions (feature mapping) φ. The feature space φ(x) could be high-dimensional, even infinite. This is good because if data aren t separable in original input space (x), they may be in feature space φ(x) We can think about how to find a good linear separator using the dot product in high dimensions, then transfer this back to kernels in the original input space.

6 Kernels and Non-linear Mappings Where does the maximization problem come from? The intuition comes from the primal version, which is based on a feature mapping φ. Theorem: Every valid kernel k(x, y) is the dot-product φ(x)[φ(x)] T for some set of basis functions (feature mapping) φ. The feature space φ(x) could be high-dimensional, even infinite. This is good because if data aren t separable in original input space (x), they may be in feature space φ(x) We can think about how to find a good linear separator using the dot product in high dimensions, then transfer this back to kernels in the original input space.

7 Why Kernels? If we can use dot products with features, why bother with kernels? Often easier to specify how similar two things are (dot product) than to construct explicit feature space φ. e.g. graphs, sets, strings (NIPS 2009 best student paper award). There are high-dimensional (even infinite) spaces that have efficient-to-compute kernels

8 Kernel Trick In previous lectures on linear models, we would explicitly compute φ(x i ) for each datapoint Run algorithm in feature space For some feature spaces, can compute dot product φ(x i ) T φ(x j ) efficiently Efficient method is computation of a kernel function k(x i, x j ) = φ(x i ) T φ(x j ) The kernel trick is to rewrite an algorithm to only have x enter in the form of dot products The menu: Kernel trick examples Kernel functions

9 Kernel Trick In previous lectures on linear models, we would explicitly compute φ(x i ) for each datapoint Run algorithm in feature space For some feature spaces, can compute dot product φ(x i ) T φ(x j ) efficiently Efficient method is computation of a kernel function k(x i, x j ) = φ(x i ) T φ(x j ) The kernel trick is to rewrite an algorithm to only have x enter in the form of dot products The menu: Kernel trick examples Kernel functions

10 Kernel Trick In previous lectures on linear models, we would explicitly compute φ(x i ) for each datapoint Run algorithm in feature space For some feature spaces, can compute dot product φ(x i ) T φ(x j ) efficiently Efficient method is computation of a kernel function k(x i, x j ) = φ(x i ) T φ(x j ) The kernel trick is to rewrite an algorithm to only have x enter in the form of dot products The menu: Kernel trick examples Kernel functions

11 Kernel Trick In previous lectures on linear models, we would explicitly compute φ(x i ) for each datapoint Run algorithm in feature space For some feature spaces, can compute dot product φ(x i ) T φ(x j ) efficiently Efficient method is computation of a kernel function k(x i, x j ) = φ(x i ) T φ(x j ) The kernel trick is to rewrite an algorithm to only have x enter in the form of dot products The menu: Kernel trick examples Kernel functions

12 A Kernel Trick Let s look at the nearest-neighbour classification algorithm For input point x i, find point x j with smallest distance: x i x j 2 = (x i x j ) T (x i x j ) = x T i x i 2x T i x j + x T j x j If we used a non-linear feature space φ( ): φ(x i ) φ(x j ) 2 = φ(x i ) T φ(x i ) 2φ(x i ) T φ(x j ) + φ(x j ) T φ(x j ) = k(x i, x i ) 2k(x i, x j ) + k(x j, x j ) So nearest-neighbour can be done in a high-dimensional feature space without actually moving to it

13 A Kernel Trick Let s look at the nearest-neighbour classification algorithm For input point x i, find point x j with smallest distance: x i x j 2 = (x i x j ) T (x i x j ) = x T i x i 2x T i x j + x T j x j If we used a non-linear feature space φ( ): φ(x i ) φ(x j ) 2 = φ(x i ) T φ(x i ) 2φ(x i ) T φ(x j ) + φ(x j ) T φ(x j ) = k(x i, x i ) 2k(x i, x j ) + k(x j, x j ) So nearest-neighbour can be done in a high-dimensional feature space without actually moving to it

14 A Kernel Trick Let s look at the nearest-neighbour classification algorithm For input point x i, find point x j with smallest distance: x i x j 2 = (x i x j ) T (x i x j ) = x T i x i 2x T i x j + x T j x j If we used a non-linear feature space φ( ): φ(x i ) φ(x j ) 2 = φ(x i ) T φ(x i ) 2φ(x i ) T φ(x j ) + φ(x j ) T φ(x j ) = k(x i, x i ) 2k(x i, x j ) + k(x j, x j ) So nearest-neighbour can be done in a high-dimensional feature space without actually moving to it

15 Example: The Quadratic Kernel Function Consider again the kernel function k(x, z) = (1 + x T z) 2 With x, z R 2, k(x, z) = (1 + x 1 z 1 + x 2 z 2 ) 2 = 1 + 2x 1 z 1 + 2x 2 z 2 + x 2 1z x 1 z 1 x 2 z 2 + x 2 2z 2 2 = (1, 2x 1, 2x 2, x 2 1, 2x 1 x 2, x 2 2)(1, 2z 1, 2z 2, z 2 1, 2z 1 z 2, z 2 2) T = φ(x) T φ(z) So this particular kernel function does correspond to a dot product in a feature space (is valid) Computing k(x, z) is faster than explicitly computing φ(x) T φ(z) In higher dimensions, larger exponent, much faster

16 Example: The Quadratic Kernel Function Consider again the kernel function k(x, z) = (1 + x T z) 2 With x, z R 2, k(x, z) = (1 + x 1 z 1 + x 2 z 2 ) 2 = 1 + 2x 1 z 1 + 2x 2 z 2 + x 2 1z x 1 z 1 x 2 z 2 + x 2 2z 2 2 = (1, 2x 1, 2x 2, x 2 1, 2x 1 x 2, x 2 2)(1, 2z 1, 2z 2, z 2 1, 2z 1 z 2, z 2 2) T = φ(x) T φ(z) So this particular kernel function does correspond to a dot product in a feature space (is valid) Computing k(x, z) is faster than explicitly computing φ(x) T φ(z) In higher dimensions, larger exponent, much faster

17 Example: The Quadratic Kernel Function Consider again the kernel function k(x, z) = (1 + x T z) 2 With x, z R 2, k(x, z) = (1 + x 1 z 1 + x 2 z 2 ) 2 = 1 + 2x 1 z 1 + 2x 2 z 2 + x 2 1z x 1 z 1 x 2 z 2 + x 2 2z 2 2 = (1, 2x 1, 2x 2, x 2 1, 2x 1 x 2, x 2 2)(1, 2z 1, 2z 2, z 2 1, 2z 1 z 2, z 2 2) T = φ(x) T φ(z) So this particular kernel function does correspond to a dot product in a feature space (is valid) Computing k(x, z) is faster than explicitly computing φ(x) T φ(z) In higher dimensions, larger exponent, much faster

18 Regression Kernelized Many classifiers can be written as using only dot products. Kernelization = replace dot products by kernel. E.g., the kernel solution for regularized least squares regression is y(x) = k(x) T (K + λi N ) 1 t vs. φ(x)(φ T Φ + λi M ) 1 Φ T t for original version N is number of datapoints (size of Gram matrix K) M is number of basis functions (size of matrix Φ T Φ) Bad if N > M, but good otherwise k(x) = (k(x, x 1,..., k(x, x n )) is the vector of kernel values over data points x n.

19 Outline Maximum Margin Criterion Math Maximizing the Margin Non-Separable Data

20 Linear Classification Consider a two class classification problem Use a linear model y(x) = w T φ(x) + b followed by a threshold function For now, let s assume training data are linearly separable Recall that the perceptron would converge to a perfect classifier for such data But there are many such perfect classifiers

21 Max Margin y = 1 y = 0 y = 1 margin We can define the margin of a classifier as the minimum distance to any example In support vector machines the decision boundary which maximizes the margin is chosen. Intuitively, this is the line right in the middle between the two classes.

22 Marginal Geometry y > 0 y = 0 y < 0 R2 x2 R1 w x y(x) w x x1 w0 w Recall from Ch. 4 y(x) = w T x + b (Using Ch.4 notation x for input rather than φ(x).) wt x w b w = y(x) w is signed distance to decision boundary.

23 Support Vectors y = 1 y = 0 y = 1 Assuming data are separated by the hyperplane, distance to decision boundary is tny(xn) w The maximum margin criterion chooses w, b by: { } 1 arg max w,b w min[t n(w T φ(x n ) + b)] n Points with this min value are known as support vectors

24 Canonical Representation This optimization problem is complex: { } 1 arg max w,b w min[t n(w T φ(x n ) + b)] n Note that rescaling w κw and b κb does not change distance tny(xn) w (many equiv. answers) So for x closest to surface, can use w w/y(x ) and b b/y(x ) so that: t (w T φ(x ) + b) = 1 All other points are at least this far away: n, t n (w T φ(x n ) + b) 1 Under these constraints, the optimization becomes: arg max w,b 1 1 = arg min w w,b 2 w 2

25 Canonical Representation This optimization problem is complex: { } 1 arg max w,b w min[t n(w T φ(x n ) + b)] n Note that rescaling w κw and b κb does not change distance tny(xn) w (many equiv. answers) So for x closest to surface, can use w w/y(x ) and b b/y(x ) so that: t (w T φ(x ) + b) = 1 All other points are at least this far away: n, t n (w T φ(x n ) + b) 1 Under these constraints, the optimization becomes: arg max w,b 1 1 = arg min w w,b 2 w 2

26 Canonical Representation This optimization problem is complex: { } 1 arg max w,b w min[t n(w T φ(x n ) + b)] n Note that rescaling w κw and b κb does not change distance tny(xn) w (many equiv. answers) So for x closest to surface, can use w w/y(x ) and b b/y(x ) so that: t (w T φ(x ) + b) = 1 All other points are at least this far away: n, t n (w T φ(x n ) + b) 1 Under these constraints, the optimization becomes: arg max w,b 1 1 = arg min w w,b 2 w 2

27 Canonical Representation This optimization problem is complex: { } 1 arg max w,b w min[t n(w T φ(x n ) + b)] n Note that rescaling w κw and b κb does not change distance tny(xn) w (many equiv. answers) So for x closest to surface, can use w w/y(x ) and b b/y(x ) so that: t (w T φ(x ) + b) = 1 All other points are at least this far away: n, t n (w T φ(x n ) + b) 1 Under these constraints, the optimization becomes: arg max w,b 1 1 = arg min w w,b 2 w 2

28 Canonical Representation So the optimization problem is now a constrained optimization problem: arg min w,b 1 2 w 2 s.t. n, t n (w T φ(x n ) + b) 1 To solve this, we need to take a detour into Lagrange multipliers

29 Outline Maximum Margin Criterion Math Maximizing the Margin Non-Separable Data

30 Lagrange Multipliers f(x) Consider the problem: g(x) x A max f (x) x s.t. g(x) = 0 g(x) = 0 Points on g(x) = 0 must have g(x) normal to surface A stationary point must have no change in f in the direction of the constraint surface, so f (x) must also be normal to the surface. So there must be some λ 0 such that f (x) + λ g(x) = 0 Define Lagrangian: L(x, λ) = f (x) + λg(x) Stationary points of L(x, λ) have x L(x, λ) = f (x) + λ g(x) = 0 and λ L(x, λ) = g(x) = 0

31 Lagrange Multipliers f(x) Consider the problem: g(x) x A max f (x) x s.t. g(x) = 0 g(x) = 0 Points on g(x) = 0 must have g(x) normal to surface A stationary point must have no change in f in the direction of the constraint surface, so f (x) must also be normal to the surface. So there must be some λ 0 such that f (x) + λ g(x) = 0 Define Lagrangian: L(x, λ) = f (x) + λg(x) Stationary points of L(x, λ) have x L(x, λ) = f (x) + λ g(x) = 0 and λ L(x, λ) = g(x) = 0

32 Lagrange Multipliers f(x) Consider the problem: g(x) x A max f (x) x s.t. g(x) = 0 g(x) = 0 Points on g(x) = 0 must have g(x) normal to surface A stationary point must have no change in f in the direction of the constraint surface, so f (x) must also be normal to the surface. So there must be some λ 0 such that f (x) + λ g(x) = 0 Define Lagrangian: L(x, λ) = f (x) + λg(x) Stationary points of L(x, λ) have x L(x, λ) = f (x) + λ g(x) = 0 and λ L(x, λ) = g(x) = 0

33 Lagrange Multipliers f(x) Consider the problem: g(x) x A max f (x) x s.t. g(x) = 0 g(x) = 0 Points on g(x) = 0 must have g(x) normal to surface A stationary point must have no change in f in the direction of the constraint surface, so f (x) must also be normal to the surface. So there must be some λ 0 such that f (x) + λ g(x) = 0 Define Lagrangian: L(x, λ) = f (x) + λg(x) Stationary points of L(x, λ) have x L(x, λ) = f (x) + λ g(x) = 0 and λ L(x, λ) = g(x) = 0

34 Lagrange Multipliers Example x 2 Consider the problem (x 1, x 2) x 1 g(x 1, x 2) = 0 max f (x 1, x 2 ) = 1 x 2 x 1 x2 2 s.t. g(x 1, x 2 ) = x 1 + x 2 1 = 0 Lagrangian: L(x, λ) = 1 x1 2 x2 2 + λ(x 1 + x 2 1) Stationary points require: L/ x 1 = 2x 1 + λ = 0 L/ x 2 = 2x 2 + λ = 0 L/ λ = x 1 + x 2 1 = 0 So stationary point is (x1, x 2 ) = ( 1 2, 1 2 ), λ = 1

35 Lagrange Multipliers - Inequality Constraints f(x) Consider the problem: g(x) > 0 xa g(x) xb g(x) = 0 max f (x) x s.t. g(x) 0 Optimization over a region solutions either at stationary points (gradients 0) in region g(x) > 0 or on boundary L(x, λ) = f (x) + λg(x) Solutions at stationary points of the Lagrangian with either: f (x) = 0 and λ = 0 (in region) f (x) = λ g(x) and λ > 0 (boundary, > for maximizing f ) For both, λg(x) = 0 Find stationary points of L s. t. g(x) 0, λ 0, λg(x) = 0

36 Lagrange Multipliers - Inequality Constraints f(x) Consider the problem: g(x) > 0 xa g(x) xb g(x) = 0 max f (x) x s.t. g(x) 0 Optimization over a region solutions either at stationary points (gradients 0) in region g(x) > 0 or on boundary L(x, λ) = f (x) + λg(x) Solutions at stationary points of the Lagrangian with either: f (x) = 0 and λ = 0 (in region) f (x) = λ g(x) and λ > 0 (boundary, > for maximizing f ) For both, λg(x) = 0 Find stationary points of L s. t. g(x) 0, λ 0, λg(x) = 0

37 Outline Maximum Margin Criterion Math Maximizing the Margin Non-Separable Data

38 Now Where Were We So the optimization problem is now a constrained optimization problem: arg min w,b w 2 s.t. n, t n (w T φ(x n ) + b) 1 For this problem, the Lagrangian (with N multipliers a n ) is: N L(w, b, a) = w 2 { a n tn (w T φ(x n ) + b) 1 } 2 n=1 We can find the derivatives of L wrt w, b and set to 0: N w = a n t n φ(x n ) 0 = n=1 2 N a n t n n=1

39 Dual Formulation Plugging those equations into L removes w and b results in a version of L where w,b L = 0: L(a) = N a n 1 2 n=1 N n=1 m=1 N a n a m t n t m φ(x n ) T φ(x m ) this new L is the dual representation of the problem (maximize with constraints) Another formula for finding b, like the bias in linear regression.

40 Outline Maximum Margin Criterion Math Maximizing the Margin Non-Separable Data

41 Non-Separable Data ξ > 1 y = 1 y = 0 y = 1 ξ < 1 ξ = 0 ξ = 0 For most problems, data will not be linearly separable (even in feature space φ) Can relax the constraints from t n y(x n ) 1 to t n y(x n ) 1 ξ n The ξ n 0 are called slack variables ξ n = 0, satisfy original problem, so x n is on margin or correct side of margin 0 < ξ n < 1, inside margin, but still correctly classifed ξ n > 1, mis-classified

42 Loss Function For Non-separable Data ξ > 1 y = 1 y = 0 y = 1 ξ < 1 ξ = 0 ξ = 0 Non-zero slack variables are bad, penalize while maximizing the margin: N min C ξ n w 2 n=1 Constant C > 0 controls importance of large margin versus incorrect (non-zero slack) Set using cross-validation Optimization is same quadratic, different constraints, convex

43 SVM Loss Function The SVM for the separable case solved the problem: 1 arg min w 2 w 2 s.t. n, t n y n 1 Can write this as: arg min w N E (t n y n 1) + λ w 2 n=1 where E (z) = 0 if z 0, otherwise Non-separable case relaxes this to be: N arg min E SV (t n y n 1) + λ w 2 w n=1 where E SV (t n y n 1) = [1 y n t n ] + hinge loss [u] + = u if u 0, 0 otherwise

44 SVM Loss Function The SVM for the separable case solved the problem: 1 arg min w 2 w 2 s.t. n, t n y n 1 Can write this as: arg min w N E (t n y n 1) + λ w 2 n=1 where E (z) = 0 if z 0, otherwise Non-separable case relaxes this to be: N arg min E SV (t n y n 1) + λ w 2 w n=1 where E SV (t n y n 1) = [1 y n t n ] + hinge loss [u] + = u if u 0, 0 otherwise

45 SVM Loss Function The SVM for the separable case solved the problem: 1 arg min w 2 w 2 s.t. n, t n y n 1 Can write this as: arg min w N E (t n y n 1) + λ w 2 n=1 where E (z) = 0 if z 0, otherwise Non-separable case relaxes this to be: N arg min E SV (t n y n 1) + λ w 2 w n=1 where E SV (t n y n 1) = [1 y n t n ] + hinge loss [u] + = u if u 0, 0 otherwise

46 Loss Functions E(z) z Linear classifiers, compare loss function used for learning z = y n t n 1 iff there is an error (with t n {+1, 1}). Black is misclassification error Transformed simple linear classifier, squared error: (y n t n ) 2 Transformed logistic regression, cross-entropy error: t n ln y n SVM, hinge loss: ξ n = [1 y n t n ] + positive only if there is a mistake, otherwise 0. encourages sparse solutions.

47 Two Views of Learning as Optimization The original SVM goal was of the form: Find the simplest hypothesis that is consistent with the data, or Maximize simplicity, given a consistency constraint. This general idea appears in much scientific model building, in image processing, and other applications. Bayesian methods use a criterion of the form Find a trade-off between simplicity and data fit, or Maximize sum of the type (data fit - λ simplicity) e.g., ln(p(d M)) λln(p(m)) where the model prior M is higher for simpler models.

48 Pros and Cons of Learning Criteria The Bayesian approach has a solid probabilistic foundation in Bayes theorem. Seems to be especially suitable for noisy data. The constraint-based approach is often easy for users to understand. Often leads to sparser simpler models. Suitable for clean data.

49 Conclusion Readings: Ch. 7 up to and including Ch Many algorithms can be re-written with only dot products of features We ve seen NN, perceptron, regression; also PCA, SVMs (later) Maximum margin criterion for deciding on decision boundary Linearly separable data Relax with slack variables for non-separable case Global optimization is possible in both cases Convex problem (no local optima) Descent methods converge to global optimum

50 Logistic Regression Learning: Iterative Reweighted Least Squares Iterative reweighted least squares (IRLS) is a descent method As in gradient descent, start with an initial guess, improve it Gradient descent - take a step (how large?) in the gradient direction IRLS is a special case of a Newton-Raphson method Approximate function using second-order Taylor expansion: ˆf (w + v) = f (w) + f (w) T (v w) (v w)t 2 f (w)(v w) Closed-form solution to minimize this is straight-forward: quadratic, derivatives linear In IRLS this second-order Taylor expansion ends up being a weighted least-squares problem, as in the linear regression case Hence the name IRLS

51 Logistic Regression Learning: Iterative Reweighted Least Squares Iterative reweighted least squares (IRLS) is a descent method As in gradient descent, start with an initial guess, improve it Gradient descent - take a step (how large?) in the gradient direction IRLS is a special case of a Newton-Raphson method Approximate function using second-order Taylor expansion: ˆf (w + v) = f (w) + f (w) T (v w) (v w)t 2 f (w)(v w) Closed-form solution to minimize this is straight-forward: quadratic, derivatives linear In IRLS this second-order Taylor expansion ends up being a weighted least-squares problem, as in the linear regression case Hence the name IRLS

52 Logistic Regression Learning: Iterative Reweighted Least Squares Iterative reweighted least squares (IRLS) is a descent method As in gradient descent, start with an initial guess, improve it Gradient descent - take a step (how large?) in the gradient direction IRLS is a special case of a Newton-Raphson method Approximate function using second-order Taylor expansion: ˆf (w + v) = f (w) + f (w) T (v w) (v w)t 2 f (w)(v w) Closed-form solution to minimize this is straight-forward: quadratic, derivatives linear In IRLS this second-order Taylor expansion ends up being a weighted least-squares problem, as in the linear regression case Hence the name IRLS

53 Logistic Regression Learning: Iterative Reweighted Least Squares Iterative reweighted least squares (IRLS) is a descent method As in gradient descent, start with an initial guess, improve it Gradient descent - take a step (how large?) in the gradient direction IRLS is a special case of a Newton-Raphson method Approximate function using second-order Taylor expansion: ˆf (w + v) = f (w) + f (w) T (v w) (v w)t 2 f (w)(v w) Closed-form solution to minimize this is straight-forward: quadratic, derivatives linear In IRLS this second-order Taylor expansion ends up being a weighted least-squares problem, as in the linear regression case Hence the name IRLS

54 Logistic Regression Learning: Iterative Reweighted Least Squares Iterative reweighted least squares (IRLS) is a descent method As in gradient descent, start with an initial guess, improve it Gradient descent - take a step (how large?) in the gradient direction IRLS is a special case of a Newton-Raphson method Approximate function using second-order Taylor expansion: ˆf (w + v) = f (w) + f (w) T (v w) (v w)t 2 f (w)(v w) Closed-form solution to minimize this is straight-forward: quadratic, derivatives linear In IRLS this second-order Taylor expansion ends up being a weighted least-squares problem, as in the linear regression case Hence the name IRLS

55 Newton-Raphson Unconstrained minimization PSfrag replacements (x, f(x)) f (x + x nt, f(x + x nt )) f Figure 9.16 The function f (shown solid) and its second-order approximation f at x (dashed). The Newton step x nt is what must be added to x to give Figure from Boyd and Vandenberghe, Convex Optimization the minimizer of f. Excellent reference, free for download online 9.5 Newton s method The Newton step For x dom f, the vector x nt = 2 f(x) 1 f(x)