LINEAR CLASSIFIERS. J. Elder CSE 4404/5327 Introduction to Machine Learning and Pattern Recognition

Transcription

1 LINEAR CLASSIFIERS

2 Classification: Problem Statement 2 In regression, we are modeling the relationship between a continuous input variable x and a continuous target variable t. In classification, the input variable x may still be continuous, but the target variable is discrete. In the simplest case, t can have only 2 values. e.g., Let t = +1 assigned to C 1 t = 1 assigned to C 2

3 Example Problem 3 Animal or Vegetable?

4 Linear Models for Classification 4 Linear models for classification separate input vectors into classes using linear (hyperplane) decision boundaries. Example: 2D Input vector x Two discrete classes C 1 and C x 2!2!4!6!8!4! x 1

5 Two Class Discriminant Function 5 y>0 y =0 y<0 x 2 R 2 R 1 y(x) = w t x +w 0 y(x)! 0 " x assigned to C 1 w x y(x) w y(x) < 0 " x assigned to C 2 Thus y(x) = 0 defines the decision boundary x w 0 w x 1

6 Two-Class Discriminant Function 6 y(x) = w t x +w 0 y(x)! 0 " x assigned to C 1 y(x) < 0 " x assigned to C 2 y>0 y =0 y<0 x 2 R 2 R 1 For convenience, let w =! w 1 w # " M $t %! w 0 w 1 w # " M $ and t w x x y(x) w x =! x 1 x # " M $t %! 1 x1 x # " M $ t w 0 w x 1 So we can express y(x) = w t x

7 Generalized Linear Models 7 For classification problems, we want y to be a predictor of t. In other words, we wish to map the input vector into one of a number of discrete classes, or to posterior probabilities that lie between 0 and 1. For this purpose, it is useful to elaborate the linear model by introducing a nonlinear activation function f, which typically will constrain y to lie between -1 and 1 or between 0 and 1. y(x) = f ( w t x + w ) 0

8 The Perceptron 8 y(x) = f ( w t x + w ) 0 y(x)! 0 " x assigned to C 1 y(x) < 0 " x assigned to C 2 A classifier based upon this simple generalized linear model is called a (single layer) perceptron. It can also be identified with an abstracted model of a neuron called the McCulloch Pitts model.

9 Parameter Learning 9 How do we learn the parameters of a perceptron?

10 Outline 10 The Perceptron Algorithm Least-Squares Classifiers Fisher s Linear Discriminant Logistic Classifiers Support Vector Machines

11 Case 1. Linearly Separable Inputs 11 For starters, let s assume that the training data is in fact perfectly linearly separable. In other words, there exists at least one hyperplane (one set of weights) that yields 0 classification error. We seek an algorithm that can automatically find such a hyperplane.

12 The Perceptron Algorithm 12 The perceptron algorithm was invented by Frank Rosenblatt (1962). The algorithm is iterative. The strategy is to start with a random guess at the weights w, and to then iteratively change the weights to move the hyperplane in a direction that lowers the classification error. Frank Rosenblatt ( )

13 The Perceptron Algorithm 13 Note that as we change the weights continuously, the classification error changes in discontinuous, piecewise constant fashion. Thus we cannot use the classification error per se as our objective function to minimize. What would be a better objective function?

14 The Perceptron Criterion 14 Note that we seek w such that w t x! 0 when t = +1 w t x < 0 when t = "1 In other words, we would like w t x n t n! 0 "n Thus we seek to minimize E P ( w) =! w t x n t n # n"m where M is the set of misclassified inputs.

15 The Perceptron Criterion 15 E P ( w) =! w t x n t n # n"m where M is the set of misclassified inputs. Observations: E P (w) is always non-negative. E P (w) is continuous and piecewise linear, and thus easier to minimize. E P ( w) w i

16 The Perceptron Algorithm 16 E P ( w) =! w t x n t n # n"m where M is the set of misclassified inputs. ( ) de P w dw =! # x t n n n"m where the derivative exists. Gradient descent: E P ( w) w! +1 = w! " #$E P ( w) = w! + # x n t n & n%m w i

17 The Perceptron Algorithm 17 w! +1 = w! " #$E P ( w) = w t + # x n t n & n%m Why does this make sense? If an input from C 1 (t = +1) is misclassified, we need to make its projection on w more positive. If an input from C 2 (t = -1) is misclassified, we need to make its projection on w more negative.

18 The Perceptron Algorithm 18 The algorithm can be implemented sequentially: Repeat until convergence: For each input (x n, t n ): If it is correctly classified, do nothing If it is misclassified, update the weight vector to be w! +1 = w! + "x n t n Note that this will lower the contribution of input n to the objective function: ( ) t (" +1) x n t n #!( w ) t x n t n =!( w ) (" ) t x n t n! $ x n t n! w (" ) ( ) t x n t n <! w (" ) ( ) t x n t n.

19 Not Monotonic 19 While updating with respect to a misclassified input n will lower the error for that input, the error for other misclassified inputs may increase. Also, new inputs that had been classified correctly may now be misclassified. The result is that the perceptron algorithm is not guaranteed to reduce the total error monotonically at each stage.

20 The Perceptron Convergence Theorem 20 Despite this non-monotonicity, if in fact the data are linearly separable, then the algorithm is guaranteed to find an exact solution in a finite number of steps (Rosenblatt, 1962).

21 Example

22 The First Learning Machine 22 Mark 1 Perceptron Hardware (c. 1960) Visual Inputs Patch board allowing configuration of inputs Rack of adaptive weights w (motor-driven potentiometers)

23 Practical Limitations 23 The Perceptron Convergence Theorem is an important result. However, there are practical limitations: Convergence may be slow If the data are not separable, the algorithm will not converge. We will only know that the data are separable once the algorithm converges. The solution is in general not unique, and will depend upon initialization, scheduling of input vectors, and the learning rate.

24 Generalization to inputs that are not linearly separable. 24 The single-layer perceptron can be generalized to yield good linear solutions to problems that are not linearly separable. Example: The Pocket Algorithm (Gal 1990) Idea: Run the perceptron algorithm Keep track of the weight vector w* that has produced the best classification error achieved so far. It can be shown that w* will converge to an optimal solution with probability 1.

25 Generalization to Multiclass Problems 25 How can we use perceptrons, or linear classifiers in general, to classify inputs when there are K > 2 classes?

26 K>2 Classes 26 Idea #1: Just use K-1 discriminant functions, each of which separates one class C k from the rest. (Oneversus-the-rest classifier.) Problem: Ambiguous regions? R 1 R 2 C 1 R 3 not C 1 C 2 not C 2

27 K>2 Classes 27 Idea #2: Use K(K-1)/2 discriminant functions, each of which separates two classes C j, C k from each other. (One-versus-one classifier.) Each point classified by majority vote. Problem: Ambiguous regions C 1 C 3 R 1 C 1? R 3 C 2 R 2 C 3 C 2

28 K>2 Classes 28 Idea #3: Use K discriminant functions y k (x) Use the magnitude of y k (x), not just the sign. y k (x) = w k t x x assigned to C k if y k (x) > y j (x)!j " k Decision boundary y k (x) = y j (x)! ( w k " w j ) t x + ( w k 0 " w j 0 ) = 0 Results in decision regions that are simply-connected and convex. R i R j R k x B x A x

29 Example: Kesler s Construction 29 The perceptron algorithm can be generalized to K- class classification problems. Example: Kesler s Construction: Allows use of the perceptron algorithm to simultaneously learn K separate weight vectors w i. Inputs are then classified in Class i if and only if w t i x > w t j x!j " i The algorithm will converge to an optimal solution if a solution exists, i.e., if all training vectors can be correctly classified according to this rule.

30 1-of-K Coding Scheme 30 When there are K>2 classes, target variables can be coded using the 1-of-K coding scheme: Input from Class C i! t = [ ] t Element i

31 Computational Limitations of Perceptrons 31 Initially, the perceptron was thought to be a potentially powerful learning machine that could model human neural processing. However, Minsky & Papert (1969) showed that the singlelayer perceptron could not learn a simple XOR function. This is just one example of a non-linearly separable pattern that cannot be learned by a single-layer perceptron. x 2 x 1 Marvin Minsky ( )

32 Multi-Layer Perceptrons 32 Minsky & Papert s book was widely misinterpreted as showing that artificial neural networks were inherently limited. This contributed to a decline in the reputation of neural network research through the 70s and 80s. However, their findings apply only to single-layer perceptrons. Multilayer perceptrons are capable of learning highly nonlinear functions, and are used in many practical applications.

33 End of Lecture 11

34 Outline 34 The Perceptron Algorithm Least-Squares Classifiers Fisher s Linear Discriminant Logistic Classifiers Support Vector Machines

35 Dealing with Non-Linearly Separable Inputs 35 The perceptron algorithm fails when the training data are not perfectly linearly separable. Let s now turn to methods for learning the parameter vector w of a perceptron (linear classifier) even when the training data are not linearly separable.

36 The Least Squares Method 36 In the least squares method, we simply fit the (x, t) observations with a hyperplane y(x). Note that this is kind of a weird idea, since the t values are binary (when K=2), e.g., 0 or 1. However it can work pretty well.

37 Least Squares: Learning the Parameters 37 Assume D! dimensional input vectors x. For each class k!1 K : y k (x) = w k t x + w k 0! y(x) = W! t!x where!x = (1,x t ) t ( ) t!w is a (D + 1)! K matrix whose kth column is!w k = w 0,w k t

38 Learning the Parameters 38 Method #2: Least Squares y(x) =! W t!x Training dataset ( x n,t n ), n = 1,,N where we use the 1-of-K coding scheme for t n Let T be the N! K matrix whose n th row is t n t Let! X be the N! (D + 1) matrix whose n th row is!x n t Let R D Setting derivative wrt! W to 0 yields: ( )!1! X t T =! X T!W =! X t! X ( W! ) = X! W!! T Then we define the error as E D ( W! ) = 1 2 2! R ij = 1 i,j 2 Tr R D { (!W ) t R! D ( W )}

39 Outline 39 The Perceptron Algorithm Least-Squares Classifiers Fisher s Linear Discriminant Logistic Classifiers Support Vector Machines

40 Fisher s Linear Discriminant 40 Another way to view linear discriminants: find the 1D subspace that maximizes the separation between the two classes. Let m 1 = 1 " x N n, m 2 = 1 " x 1 N n 2 n!c 1 n!c 2 For example, might choose w to maximize w t ( m 2! m 1 ), subject to w = 1 This leads to w! m 2 " m However, if conditional distributions are not isotropic, this is typically not optimal. 0!2!2 2 6

41 Fisher s Linear Discriminant 41 Let m 1 = w t m 1, m 2 = w t m 2 be the conditional means on the 1D subspace. Let s k 2 = # ( y n! m k ) 2 be the within-class variance on the subspace for class C k n"c k ( The Fisher criterion is then J(w) = m! m 2 1) 2 s s 2 This can be rewritten as J(w) = w t S B w w t S W w where S B = m 2! m 1 and ( )( m 2! m 1 ) t is the between-class variance S W = # x n! m 1 + # x n! m 2 is the within-class variance n"c 1 ( )( x n! m 1 ) t n"c 2 J(w) is maximized for w! S "1 W ( m 2 " m 1 ) ( )( x n! m 2 ) t 4 2 0!2!2 2 6

42 42 Connection between Least-Squares and FLD Change coding scheme used in least-squares method to t n = N N 1 for C 1 t n =! N N 2 for C 2 Then one can show that the ML w satisfies ( ) w! S W "1 m 2 " m 1

43 Least Squares Classifier 43 Problem #1: Sensitivity to outliers !2!2!4!4!6!6!8!4! !8!4!

44 Least Squares Classifier 44 Problem #2: Linear activation function is not a good fit to binary data. This can lead to problems !2!4!6!6!4!

45 Outline 45 The Perceptron Algorithm Least-Squares Classifiers Fisher s Linear Discriminant Logistic Classifiers Support Vector Machines

46 Logistic Regression (K = 2) 46 ( ) p( C 1! ) = y (!) = " w t! p( C 2! ) = 1# p( C 1! ) p ( C 1! ) = y (!) = " ( w t!) where! (a) =!" 1 1+ exp("a) w t! x 1

47 Logistic Regression 47 ( ) = y (!) = " w t! ( ) = 1# p ( C 1! ) p C 1! p C 2! where! (a) = 1 1+ exp("a) ( ) Number of parameters Logistic regression: M Gaussian model: 2M + 2M(M+1)/2 + 1 = M 2 +3M+1

48 ML for Logistic Regression 48 ( ) t = y n { n 1! y } 1!t n n p t w N " where t = t 1,,t N n=1 ( ) t and y n = p ( C 1! ) n We define the error function to be E(w) =! log p ( t w) Given y n =! ( a ) n and a n = w t " n, one can show that!e(w) = N $ ( y n " t n )# n n=1 Unfortunately, there is no closed form solution for w.

49 End of Lecture 12

50 ML for Logistic Regression: 50 Iterative Reweighted Least Squares Although there is no closed form solution for the ML estimate of w, fortunately, the error function is convex. Thus an appropriate iterative method is guaranteed to find the exact solution. A good method is to use a local quadratic approximation to the log likelihood function (Newton- Raphson update): w (new) = w (old )! H!1 "E(w) where H is the Hessian matrix of E(w)

51 ML for Logistic Regression 51 w (new) = w (old )! H!1 "E(w) where H is the Hessian matrix of E(w) : H =! t R! where R is the N " N diagonal weight matrix with R nn = y n ( 1# y n ) (Note that, since R nn! 0, R is positive semi-definite, and hence H is positive semi-definite Thus E(w) is convex.) Thus ( ) w new = w (old )! (" t R" )!1 " t y! t

52 ML for Logistic Regression 52 Iterative Reweighted Least Squares ( ) = y (!) = " w t! p C 1! ( )!! # " w 1 " # w 2 w t!

53 Logistic Regression 53 For K>2, we can generalize the activation function by modeling the posterior probabilities as p( C k! ) = y k (!) = exp a k " ( ) ( ) exp a j where the activations a k are given by a k = w k t! j

54 Example !2 2!4 4!6!6!4! Least-Squares Logistic

55 Outline 55 The Perceptron Algorithm Least-Squares Classifiers Fisher s Linear Discriminant Logistic Classifiers Support Vector Machines

56 Motivation 56 The perceptron algorithm is guaranteed to provide a linear decision surface that separates the training data, if one exists. However, if the data are linearly separable, there are in general an infinite number of solutions, and the solution returned by the perceptron algorithm depends in a complex way on the initial conditions, the learning rate and the order in which training data are processed. While all solutions achieve a perfect score on the training data, they won t all necessarily generalize as well to new inputs.

57 Which solution would you choose? 57

58 The Large Margin Classifier 58 Unlike the Perceptron Algorithm, Support Vector Machines solve a problem that has a unique solution: they return the linear classifier with the maximum margin, that is, the hyperplane that separates the data and is farthest from any of the training vectors. Why is this good?

59 Support Vector Machines 59 SVMs are based on the linear model y(x) = w t!(x) + b Assume training data x 1,,x N with corresponding target values t 1,,t N, t n!{"1,1}. x classified according to sign of y(x). Assume for the moment that the training data are linearly separable in feature space. Then!w,b : t n y ( x n ) > 0 "n #[1, N]

60 Maximum Margin Classifiers 60 When the training data are linearly separable, there are generally an infinite number of solutions for (w, b) that separate the classes exactly. The margin of such a classifier is defined as the orthogonal distance in feature space between the decision boundary and the closest training vector. SVMs are an example of a maximum margin classifer, which finds the linear classifier that maximizes the margin. y =1 y =0 y = 1 margin

61 Probabilistic Motivation 61 The maximum margin classifier has a probabilistic motivation. If we model the class-conditional densities with a KDE using Gaussian kernels with variance! 2, then in the limit as! " 0, the optimal linear decision boundary " maximum margin linear classifier. y =1 y =0 y = 1 margin

62 Two Class Discriminant Function 62 Recall: y>0 y =0 y<0 x 2 R 2 R 1 y(x) = w t x +w 0 y(x)! 0 " x assigned to C 1 w x y(x) w y(x) < 0 " x assigned to C 2 Thus y(x) = 0 defines the decision boundary x w 0 w x 1

63 Maximum Margin Classifiers 63 Distance of point x n from decision surface is given by: t n y x ( t n! ( x n ) + b) ( ) w = t n w w Thus we seek: argmax w,b &( ' )( 1 w min n " # t n w t! ( x n ) + b ( ) * $ ( % +,( y =1 y =0 y = 1 margin

64 Maximum Margin Classifiers 64 Distance of point x n from decision surface is given by: t n y x ( t n! ( x n ) + b) ( ) w = t n w w Note that rescaling w and b by the same factor leaves the distance to the decision surface unchanged. Thus, wlog, we consider only solutions that satisfy: t n ( w t! ( x n ) + b) = 1. for the point x n that is closest to the decision surface. y =1 y =0 y = 1 margin

65 Quadratic Programming Problem 65 Then all points x n satisfy t n ( w t! ( x n ) + b) " 1 Points for which equality holds are said to be active. All other points are inactive. Now argmax w,b &( ' )( argmin w 2 1 w min n " # t n w t! ( x n ) + b ( ) * $ ( % +,( y =1 y =0 y = 1 w Subject to t n ( w t! ( x n ) + b). 1 /x n This is a quadratic programming problem. margin Solving this problem will involve Lagrange multipliers.

66 Lagrange Multipliers

67 Lagrange Multipliers (Appendix C.4 in textbook) 67 Used to find stationary points of a function subject to one or more constraints. Example (equality constraint): f(x) Maximize f ( x) subject to g( x) = 0. g(x) x A Observations: g(x) =0 Joseph-Louis Lagrange At any point on the constraint surface,!g( x) must be orthogonal to the surface. 2. Let x * be a point on the constraint surface where f (x) is maximized. Then!f (x) is also orthogonal to the constraint surface. 3.! "# $ 0 such that %f (x) + #%g(x) = 0 at x *. # is called a Lagrange multiplier.

68 Lagrange Multipliers (Appendix C.4 in textbook) 68!" # 0 such that $f (x) + "$g(x) = 0 at x *. Defining the Lagrangian function as: ( ) = f (x) +!g(x) L x,! we then have and ( ) = 0.! x L x," ( )!"!L x," = 0. g(x) x A f(x) g(x) =0

69 Example 69 L( x,! ) = f (x) +!g(x) x 2 Find the stationary point of ( ) = 1! x 1 2! x 2 2 f x 1,x 2 (x 1,x 2) x 1 subject to g(x 1,x 2 )=0 g( x 1,x ) 2 = x 1 + x 2! 1= 0

70 End of Lecture 13

71 71 Inequality Constraints Maximize f ( x) subject to g( x)! 0. There are 2 cases: 1. x* on the interior (e.g., x B ) g(x) x A f(x) Here g(x) > 0 and the stationary condition is simply This corresponds to a stationary point of the Lagrangian where = x* on the boundary (e.g., x A )!f (x) = 0. Here g(x) = 0 and the stationary condition is!f (x) = "#!g(x), # > 0. This corresponds to a stationary point of the Lagrangian where > 0. g(x) > 0 x B g(x) =0 L( x,! ) = f (x) +!g(x) Thus the general problem can be expressed as maximizing the Lagrangian subject to 1. g(x)! 0 2. "! 0 Karush-Kuhn-Tucker (KKT) conditions 3. "g(x) = 0

72 Minimizing vs Maximizing 72 If we want to minimize f(x) subject to g(x) 0, then the Lagrangian becomes L( x,! ) = f (x) "!g(x) with! # 0.

73 Extension to Multiple Constraints 73 Suppose we wish to maximize f(x) subject to g j (x) = 0 for j = 1,,J h k (x)! 0 for k = 1,,K We then find the stationary points of J ( ) = f (x) +! j g j (x) L x,! subject to h k (x)! 0 µ k! 0 µ k h k (x) = 0 " + " µ k h k (x) j=1 K k=1

74 Back to our quadratic programming problem

75 Quadratic Programming Problem argmin w 2 w, subject to t ( n w t! ( x n ) + b) " 1 #x n Solve using Lagrange multipliers a n : L(w,b,a) = 1 2 w 2! a n t n w t " x n N # n=1 ( ( ) + b)! 1 { } y =1 y =0 y = 1 margin

76 Dual Representation 76 Solve using Lagrange multipliers a n : L(w,b,a) = 1 2 w 2! a n t n w t " x n N # n=1 ( ( ) + b)! 1 { } Setting derivatives with respect to w and b to 0, we get: N " w = a n t n!(x n ) N n=1 " a n t n = 0 n=1 y =1 y =0 y = 1 margin

77 Dual Representation 77 L(w,b,a) = 1 2 w 2! a n t n w t " x n N # n=1 ( ( ) + b)! 1 { } w = a n t n!(x n ) N N " n=1 " a n t n = 0 n=1 Substituting leads to the dual representation of the maximum margin problem, in which we maximize: N ( ) = a n!l a! " 1 2 n=1 N N!! n=1 m=1 with respect to a, subject to: a n # 0 $n N! a n t n = 0 n=1 and where k x, x% ( ) = &(x) t &( % a n a m t n t m k ( x n,x m ) x ) margin y =1 y =0 y = 1

78 Dual Representation 78 N Using w = " a n t n!(x n ), a new point x is classified by computing N n=1 y(x) = " a n t n k(x,x n ) + b n=1 The Karush-Kuhn-Tucker (KKT) conditions for this constrained optimization problem are: a n! 0 ( ) " 1! 0 t n y ( x n ) " 1 t n y x n a n { } = 0 ( ) = 1. Thus for every data point, either a n = 0 or t n y x n support vectors y = 1 y =0 y =1

79 Solving for the Bias 79 Once the optimal a is determined, the bias b can be computed by noting that any support vector x n satisfies t n y x n Using y(x) =! a n t n k(x,x n ) + b ( ) = 1. A more numerically accurate solution can be obtained by averaging over all support vectors: b = 1 $ ' # t N n! # a m t m k(x n,x m ) S % & ( ) n"s N n=1 N " % we have t n! a m t m k(x n,x m ) + b # $ & ' = 1 m=1 and so b = t n! a m t m k(x n,x m ) N " m=1 m"s where S is the index set of support vectors and N S is the number of support vectors.

80 Example (Gaussian Kernel) 80 Input Space x 2 x 1

81 Overlapping Class Distributions 81 The SVM for non-overlapping class distributions is determined by solving 1 2 argmin w 2 w N # E! y x n + $ w 2 n=1, subject to t n ( w t! ( x n ) + b) " 1 #x n Alternatively, this can be expressed as the minimization of ( ( )t n " 1) where E! (z) is 0 if z % 0, and! otherwise. ξ > 1 y = 1 y =0 y =1 This forces all points to lie on or outside the margins, on the correct side for their class. To allow for misclassified points, we have to relax this E! term. ξ < 1 ξ =0 ξ =0

82 Slack Variables 82 To this end, we introduce N slack variables! n " 0, n = 1, N.! n = 0 for points on or on the correct side of the margin boundary for their class! n = t n " y ( x n ) for all other points. Thus! n < 1 for points that are correctly classified! n > 1 for points that are incorrectly classified We now minimize C "! n w 2, where C > 0. subject to t n y x n N n=1 ( )! 1" # n, and # n! 0, n = 1, N ξ > 1 y = 1 y =0 y =1 ξ < 1 ξ =0 ξ =0

83 Dual Representation 83 This leads to a dual representation, where we maximize!l(a) = N! a n " 1 n=1 2 with constraints 0 # a n # C and N! a n t n = 0 n=1 N N!! a n a m t n t m k ( x n,x ) n n=1 m=1 ξ < 1 ξ > 1 y = 1 y =0 ξ =0 y =1 ξ =0

84 Support Vectors 84 Again, a new point x is classified by computing N y(x) =! a n t n k(x,x n ) + b n=1 For points that are on the correct side of the margin, a n = 0. Thus support vectors consist of points between their margin and the decision boundary, as well as misclassified points. In other words, all points that are not on the right side of their margin are support vectors. ξ > 1 y = 1 y =0 y =1 ξ < 1 ξ =0 ξ =0

85 Bias 85 Again, a new point x is classified by computing N y(x) =! a n t n k(x,x n ) + b n=1 Once the optimal a is determined, the bias b can be computed from b = 1 $ ' # t N n! # a m t m k(x n,x m ) M % & ( ) n"m m"s where S is the index set of support vectors N S is the number of support vectors M is the index set of points on the margins N M is the number of points on the margins ξ > 1 ξ < 1 y = 1 y =0 y =1 ξ =0 ξ =0

86 86 Solving the Quadratic Programming Problem Maximize! L(a) = N! a n " 1 n=1 2 Problem is convex. n=1 m=1 a n a m t n t m k ( x n,x m ) Standard solutions are generally O(N 3 ). N!! subject to 0 # a n # C and! a n t n = 0 N N n=1 Traditional quadratic programming techniques often infeasible due to computation and memory requirements. Instead, methods such as sequential minimal optimization can be used, that in practice are found to scale as O(N) - O(N 2 ).

87 Chunking 87 Maximize! L(a) = N! a n " 1 n=1 2 Conventional quadratic programming solution requires that matrices with N 2 elements be maintained in memory. K! O( N 2 ), where K nm = k x n,x m T! O( N 2 ), where T nm = t n t m A! O( N 2 ), where A nm = a n a m N!! n=1 m=1 subject to 0 # a n # C and! a n t n = 0 N N n=1 a n a m t n t m k ( x n,x m ) ( ) This becomes infeasible when N exceeds ~10,000.

88 Chunking 88 Maximize! L(a) = N! a n " 1 n=1 2 N!! n=1 m=1 subject to 0 # a n # C and! a n t n = 0 N N n=1 a n a m t n t m k ( x n,x m ) Chunking (Vapnik, 1982) exploits the fact that the value of the Lagrangian is unchanged if we remove the rows and columns of the kernel matrix where a n = 0 or a m = 0.

89 Chunking 89 N Minimize C "! n w 2, where C > 0. n=1! n = 0 for points on or on the correct side of the margin boundary for their class! n = t n " y ( x n ) for all other points. Chunking (Vapnik, 1982) 1. Select a small number (a chunk ) of training vectors 2. Solve the QP problem for this subset 3. Retain only the support vectors 4. Consider another chunk of the training data 5. Ignore the subset of vectors in all chunks considered so far that lie on the correct side of the margin, since these do not contribute to the cost function 6. Add the remainder to the current set of support vectors and solve the new QP problem 7. Return to Step 4 8. Repeat until the set of support vectors does not change. 2 This method reduces memory requirements to O( N S ), where N S is the number of support vectors. This may still be big!

90 Decomposition Methods 90 It can be shown that the global QP problem is solved when, for all training vectors, satisfy the following optimality conditions: a i = 0! t i y x i ( ) " 1. ( ) = 1. ( ) # 1. 0 < a i < C! t i y x i a i = C! t i y x i Decomposition methods decompose this large QP problem into a series of smaller subproblems. Decomposition (Osuna et al, 1997) Partition the training data into a small working subset B and a fixed subset N. Minimize the global objective function by adjusting the coefficients in B Swap 1 or more vectors in B for an equal number in N that fail to satisfy the optimality conditions Re-solve the global QP problem for B Each step is O(B) 2 in memory. Osuna et al (1997) proved that the objective function decreases on each step and will converge in a finite number of iterations.

91 Sequential Minimal Optimization 91 Sequential Minimal Optimization (Platt 1998) takes decomposition to the limit. On each iteration, the working set consists of just two vectors. The advantage is that in this case, the QP problem can be solved analytically. Memory requirement are O(N). Compute time is typically O(N) O(N 2 ).

92 LIBSVM 92 LIBSVM is a widely used library for SVMs developed by Chang & Lin (2001). Can be downloaded from MATLAB interface Uses SMO Will use for Assignment 2.

93 End of Lecture 14

94 LIBSVM Example: Face Detection 94 Face Preprocess: Subsample & Normalize µ = 0,! 2 = 1 svmtrain Non-Face Preprocess: Subsample & Normalize µ = 0,! 2 = 1

95 LIBSVM Example: MATLAB Interface 95 Selects linear SVM model=svmtrain(traint, trainx, '-t 0'); [predicted_label, accuracy, decision_values] = svmpredict(testt, testx, model); Accuracy = % (661/944) (classification)

96 Example 96 Input Space 2 x 2 0!2!2 0 2 x 1

97 Relation to Logistic Regression 97 The objective function for the soft-margin SVM can be written as: ( ) N! E SV y n t n + " w 2 n=1 where E SV and $% z & ' + = z if z ( 0 = 0 otherwise. ( z) = $% 1# z& '+ is the hinge error function, For t!{"1,1}, the objective function for a regularized version of logistic regression can be written as: ( ) N # E LR y n t n + $ w 2 n=1 where E LR ( z) = log( 1+ exp("z) ). E(z) E LR E SV z