Lecture 3: Multiclass Classification

Transcription

1 Lecture 3: Multiclass Classification Kai-Wei Chang University of Virginia kw@kwchang.net Some slides are adapted from Vivek Skirmar and Dan Roth CS6501 Lecture 3 1

2 Announcement v Please enroll in Piazza and Collab v Makeup class for 2/8 (Wed) (2/10 or 2/17) CS6501 Lecture 3 2

3 Previous Lecture v Binary linear classification models v Perceptron, SVMs, Logistic regression v Prediction is simple: v Given an example x, prediction is sgn w & x v Note that all these linear classifier have the same inference rule v In logistic regression, we can further estimate the probability v Question? CS6501 Lecture 3 3

4 This Lecture v Multiclass classification overview v Reducing multiclass to binary v One-against-all & One-vs-one v Error correcting codes v Training a single classifier v Multiclass Perceptron: Kesler s construction v Multiclass SVMs: Crammer&Singer formulation v Multinomial logistic regression CS6501 Lecture 3 4

5 What is multiclass v Output 1,2,3, K v In some cases, output space can be very large (i.e., K is very large) v Each input belongs to exactly one class (c.f. in multilabel, input belongs to many classes) CS6501 Lecture 3 5

6 Example applications CS6501 Lecture 3 6

7 Two key ideas to solve multiclass v Reducing multiclass to binary v Decompose the multiclass prediction into multiple binary decisions v Make final decision based on multiple binary classifiers v Training a single classifier v Minimize the empirical risk v Consider all classes simultaneously CS6501 Lecture 3 7

8 Reduction v.s. single classifier v Reduction v Future-proof: binary classification improved so does muti-class v Easy to implement v Single classifier v Global optimization: directly minimize the empirical loss; easier for joint prediction v Easy to add constraints and domain knowledge CS6501 Lecture 3 8

9 A General Formula y0 = argmax y Y f(y; w, x) input model parameters output space v Inference/Test: given w, x, solve argmax v Learning/Training: find a good w v Today: x R n, Y = {1,2, K} (multiclass) CS6501 Lecture 3 9

10 This Lecture v Multiclass classification overview v Reducing multiclass to binary v One-against-all & One-vs-one v Error correcting codes v Training a single classifier v Multiclass Perceptron: Kesler s construction v Multiclass SVMs: Crammer&Singer formulation v Multinomial logistic regression CS6501 Lecture 3 10

11 One against all strategy CS6501 Lecture 3 11

12 One against All learning v Multiclass classifier v Function f : R n à {1,2,3,...,k} v Decompose into binary problems CS6501 Lecture 3 12

13 One-again-All learning algorithm v Learning: Given a dataset D = x C R E, y C 1,2,3, K x C,y C v Decompose into K binary classification tasks v Learn K models: w F, w G, w H, w I v For class k, construct a binary classification task as: v Positive examples: Elements of D with label k v Negative examples: All other elements of D v The binary classification can be solved by any algorithm we have seen CS6501 Lecture 3 13

14 One against All learning v Multiclass classifier v Function f : R n à {1,2,3,...,k} v Decompose into binary problems Ideal case: only the correct label will have a positive score & & & w JKLMN x > 0 w JKRS x > 0 w TUSSE x > 0 CS6501 Lecture 3 14

15 One-again-All Inference v Learning: Given a dataset D = x C,y C x C R E, y C 1,2,3, K v Decompose into K binary classification tasks v Learn K models: w F, w G, w H, w I v Inference: Winner takes all v y0 = argmax V {F,G, I} w& V x & For example: y = argmax (w JKLMN v An instance of the general form y0 = argmax y Y f(y; w, x) & x, w JKRS x, w& TUSSE x) w = {w F, w G, w I }, f y; w, x = w V & x CS6501 Lecture 3 15

16 One-again-All analysis v Not always possible to learn v Assumption: each class individually separable from all the others v No theoretical justification v Need to make sure the range of all classifiers is the same we are comparing scores produced by K classifiers trained independently. v Easy to implement; work well in practice CS6501 Lecture 3 16

17 One v.s. One (All against All) strategy CS6501 Lecture 3 17

18 One v.s. One learning v Multiclass classifier v Function f : R n à {1,2,3,...,k} v Decompose into binary problems Training Test CS6501 Lecture 3 18

19 One-v.s-One learning algorithm v Learning: Given a dataset D = x C R E, y C 1,2,3, K x C,y C v Decompose into C(K,2) binary classification tasks v Learn C(K,2) models: w F, w G, w H, w I (IYF)/G v For each class pair (i,j), construct a binary classification task as: v Positive examples: Elements of D with label i v Negative examples Elements of D with label j v The binary classification can be solved by any algorithm we have seen CS6501 Lecture 3 19

20 One-v.s-One Inference algorithm v Decision Options: v More complex; each label gets k-1 votes v Output of binary classifier may not cohere. v Majority: classify example x to take label i if i wins on x more often than j (j=1, k) v A tournament: start with n/2 pairs; continue with winners CS6501 Lecture 3 20

21 Classifying with One-vs-one Tournament Majority Vote 1 red, 2 yellow, 2 green è? All are post-learning and might cause weird stuff CS6501 Lecture 3 21

22 One-v.s.-one Assumption v Every pair of classes is separable It is possible to separate all k classes with the O(k 2 ) classifiers Decision Regions 22 CS6501 Lecture 3

23 Comparisons v One against all v O(K) weight vectors to train and store v Training set of the binary classifiers may unbalanced v Less expressive; make a strong assumption v One v.s. One (All v.s. All) v O(K G ) weight vectors to train and store v Size of training set for a pair of labels could be small overfitting of the binary classifiers v Need large space to store model CS6501 Lecture 3 23

24 Problems with Decompositions v Learning optimizes over local metrics v Does not guarantee good global performance v We don t care about the performance of the local classifiers v Poor decomposition poor performance v Difficult local problems v Irrelevant local problems v Efficiency: e.g., All vs. All vs. One vs. All v Not clear how to generalize multi-class to problems with a very large # of output CS6501 Lecture 3 24

25 Still an ongoing research direction Key questions: v How to deal with large number of classes v How to select right samples to train binary classifiers v Error-correcting tournaments [Beygelzimer, Langford, Ravikumar 09] v Logarithmic Time One-Against-Some [Daume, Karampatziakis, Langford, Mineiro 16] v Label embedding trees for large multi-class tasks. [Bengio, Weston, Grangier 10] v CS6501 Lecture 3 25

26 Decomposition methods: Summary v General Ideas: v Decompose the multiclass problem into many binary problems v Prediction depends on the decomposition v Constructs the multiclass label from the output of the binary classifiers v Learning optimizes local correctness v Each binary classifier don t need to be globally correct and isn t aware of the prediction procedure CS6501 Lecture 3 26

27 This Lecture v Multiclass classification overview v Reducing multiclass to binary v One-against-all & One-vs-one v Error correcting codes v Training a single classifier v Multiclass Perceptron: Kesler s construction v Multiclass SVMs: Crammer&Singer formulation v Multinomial logistic regression CS6501 Lecture 3 27

28 Revisit One-again-All learning algorithm v Learning: Given a dataset D = x C R E, y C 1,2,3, K x C,y C v Decompose into K binary classification tasks v Learn K models: w F, w G, w H, w I v w N : separate class k from others v Prediction y0 = argmax V {F,G, I} w V & x CS6501 Lecture 3 28

29 Observation v At training time, we require w C & x to be positive for examples of class i. v Really, all we need is for w C & x to be more than all others this is a weaker requirement For examples with label i, we need w & C x > w & _ x for all j CS6501 Lecture 3 29

30 Perceptron-style algorithm v For each training example x, y For examples with label i, we need w & C x > w & _ x for all j v If for some y, w & V x w & Vb x mistake! v w V w V + ηx update to promote y v w Vb w Vb ηx update to demote y Why add ηx to w V promote label y: Before update s y = < w V ikj,x > After update s y =< w V ESk, x >=< w V ikj +ηx, x > = < w V ikj,x > +η < x, x > Note! < x, x > = x & x > 0 CS6501 Lecture 3 30

31 A Perceptron-style Algorithm Given a training set D = { x, y } Initialize w 0 R E For epoch 1 T: How to analyze this algorithm For (x,y) in D: For y b and simplify the update rules? y if w & V x < w & Vb x make a mistake w V w V + ηx promote y w Vb w Vb ηx demote y Return w Prediction: argmax q w V & x CS6501 Lecture 3 31

32 Linear Separability with multiple classes v Let s rewrite the equation w C & x > w _ & x for all j v Instead of having w F, w G, w H, w I, we want to represent the model using a single vector w w &? > w &? for all j v How? Change the input representation Let s define φ(x, y), such that w & φ x, i > w & φ x, j j multiple models v.s. multiple data points CS6501 Lecture 3 32

33 Kesler construction Assume we have a multi-class problem with K class and n features. w C & x > w _ & x j v models: w F, w G, w I, v Input: x R E w N R E w & φ x, i > w & φ x, j j CS6501 Lecture 3 33

34 Kesler construction Assume we have a multi-class problem with K class and n features. w C & x > w _ & x j v models: w F, w G, w I, v Input: x R E w N R E w & φ x, i > w & φ x, j j v Only one model: w R E I CS6501 Lecture 3 34

35 Kesler construction Assume we have a multi-class problem with K class and n features. w C & x > w _ & x j v models: w F, w G, w I, v Input: x R E w N R E w & φ x, i > w & φ x, j j v Only one model: w R EI v Define φ x, y for label y being associated to input x 0 E φ x, y = x 0 E EI F x in y }~ block; Zeros elsewhere CS6501 Lecture 3 35

36 Kesler construction Assume we have a multi-class problem with K class and n features. w C & x > w C & x i v models: w F, w G, w I, w N R E v Input: x R E w = w & φ x, i > w & φ x, j j w F w V w E EI F φ x, y = 0 E x 0 E EI F x in y }~ block; Zeros elsewhere w & φ x, y = w V & x CS6501 Lecture 3 36

37 Kesler construction Assume we have a multi-class problem with K class and n features. w & φ x, i > w & φ x, j j w & φ x, i φ x, j > 0 j binary classification problem w = w F w V [φ x, i φ x, j ] = w E EI F 0 E x in i }~ block; x x in j }~ block; x 0 E EI F CS6501 Lecture 3 37

38 Linear Separability with multiple classes What we want w & φ x, i > w & φ x, j j w & φ x, i φ x, j > 0 j For all example (x, y) with all other labels y in dataset, w in nk dimension should linearly separate φ x, i φ x, j and [φ x, i φ x, j ] CS6501 Lecture 3 38

39 How can we predict? argmax q w & φ(x, y) For input an input x, the model predict label is 3 w = w F w V w E EI F φ x, y = 0 E x 0 E EI F φ(x, 3) w φ(x, 2) φ(x, 1) φ(x, 4) φ(x, 5) CS6501 Lecture 3 39

40 How can we predict? w = argmax q w & φ(x, y) w F w V w E EI F φ x, y = 0 E x 0 E EI F w For input an input x, the model predict label is 3 φ(x, 3) φ(x, 2) φ(x, 1) φ(x, 4) φ(x, 5) This is equivalent to argmax V {F,G, I} w V & x CS6501 Lecture 3 40

41 Constraint Classification v Goal: w φ x, i φ x, j 0 j v Training: v For each example x, i v Update model if w & φ x, i φ x, j < 0, j x Transform Examples -x 2>4 2>1 2>1 2>3 2>4 2>3 CS6501 Lecture 3 41

42 A Perceptron-style Algorithm Given a training set D = { x, y } Initialize w 0 R E For epoch 1 T: For (x,y) in D: This needs D K updates, For y b do we need all of them? y if w & φ x, y φ x, y < 0 Return w w w + η φ x, y φ x, y How to interpret this update rule? Prediction: argmax q w & φ(x, y) CS6501 Lecture 3 42

43 An alternative training algorithm v Goal: v Training: w φ x, i φ x, j w & φ x, i max _ C w & φ x, i max _ v For each example x, i v Find the prediction of the current model: y0 = argmax w & φ(x, j) v Update model if w & φ x, i φ x, y0 0 j w & φ x, i 0 w & φ x, i 0 < 0, y CS6501 Lecture 3 43

44 A Perceptron-style Algorithm Given a training set D = { x, y } Initialize w 0 R E For epoch 1 T: For (x,y) in D: Return w y = argmax qb w & φ(x, y ) if w & φ x, y φ x, y0 < 0 w w + η φ x, y φ x, y How to interpret this update rule? Prediction: argmax q w & φ(x, y) CS6501 Lecture 3 44

45 A Perceptron-style Algorithm Given a training set D = { x, y } Initialize w 0 R E There are only two situations: For epoch 1 T: 1. y0 = y: φ x, y φ x, y0 =0 For (x,y) in D: 2. w & φ x, y φ x, y0 < 0 Return w y = argmax qb w & φ(x, y ) if w & φ x, y φ x, y0 < 0 w w + η φ x, y φ x, y0 How to interpret this update rule? Prediction: argmax q w & φ(x, y) CS6501 Lecture 3 45

46 A Perceptron-style Algorithm Given a training set D = { x, y } Initialize w 0 R E For epoch 1 T: For (x,y) in D: Return w y = argmax qb w & φ(x, y ) w w + η φ x, y φ x, y0 How to interpret this update rule? Prediction: argmax q w & φ(x, y) CS6501 Lecture 3 46

47 Consider multiclass margin CS6501 Lecture 3 47

48 Marginal constraint classifier v Goal: for every (x,y) in the training data set min V V w& φ x, y φ x, y δ w & φ x, y max V Vb w& φ x, y δ w & φ x, i [max V Vb w& φ x, j + δ] 0 δ δ δ Constraints violated need an update Let s define: Δ y, y b δ if y y = 0 if y = y Check if y = argmax V w & φ x, y + Δ(y, y b ) CS6501 Lecture 3 48

49 A Perceptron-style Algorithm Given a training set D = { x, y } Initialize w 0 R E For epoch 1 T: For (x,y) in D: Return w y = argmax qb w & φ x, y + Δ(y, y ) w w + η φ x, y φ x, y0 δ δ δ How to interpret this update rule? Prediction: argmax q w & φ(x, y) CS6501 Lecture 3 49

50 Remarks v This approach can be generalized to train a ranker; in fact, any output structure v We have preference over label assignments v E.g., rank search results, rank movies / products CS6501 Lecture 3 50

51 A peek of a generalized Perceptron model Given a training set D = { x, y } Initialize w 0 R E Structured output For epoch 1 T: For (x,y) in D: Structural prediction/inference Return w y = argmax qb w & φ(x, y ) + Δ(y, y ) w w + η φ x, y φ x, y0 Model update Structural loss Prediction: argmax q w & φ(x, y) CS6501 Lecture 3 51

52 Feb 1. CS6501 Lecture 3 52

53 Recap: A Perceptron-style Algorithm Given a training set D = { x, y } Initialize w 0 R E For epoch 1 T: For (x,y) in D: For y b y if w & V x < w & Vb x make a mistake promote y w V w V + ηx w Vb w Vb ηx demote y Return w Prediction: argmax q w V & x CS6501 Lecture 3 53

54 Recap: Kesler construction Assume we have a multi-class problem with K class and n features. w C & x > w C & x i v models: w F, w G, w I, w N R E v Input: x R E w = w & φ x, i > w & φ x, j j w F w V w E EI F φ x, y = 0 E x 0 E EI F x in y }~ block; Zeros elsewhere w & φ x, y = w V & x CS6501 Lecture 3 54

55 Geometric interpretation # features = n; # classes = K In R Ÿ space In R ŸI space w G w F w φ(x, 3) φ(x, 2) φ(x, 4) w φ(x, 1) w H w φ(x, 5) argmax V {F,G, I} w V & x argmax q w & φ(x, y) CS6501 Lecture 3 55

56 Recap: A Perceptron-style Algorithm Given a training set D = { x, y } Initialize w 0 R E For epoch 1 T: For (x,y) in D: Return w y = argmax qb w & φ(x, y ) w w + η φ x, y φ x, y0 How to interpret this update rule? Prediction: argmax q w & φ(x, y) CS6501 Lecture 3 56

57 Multi-category to Constraint Classification v Multiclass v (x, A) (x, (A>B, A>C, A>D) ) v Multilabel v (x, (A, B)) (x, ( (A>C, A>D, B>C, B>D) ) v Label Ranking v (x, (5>4>3>2>1)) (x, ( (5>4, 4>3, 3>2, 2>1) ) CS6501 Lecture 3 57

58 Generalized constraint classifiers v In all cases, we have examples (x,y) with y S k v Where S k : partial order over class labels {1,...,k} v defines preference relation ( > ) for class labeling v Consequently, the Constraint Classifier is: h: X S k v h(x) is a partial order v h(x) is consistent with y if (i<j) y è (i<j) h(x) Just like in the multiclass we learn one w i R n for each label, the same is done for multi-label and ranking. The weight vectors are updated according with the requirements from y S k CS6501 Lecture 3 58

59 Multi-category to Constraint Classification Solving structured prediction problems by ranking algorithms v Multiclass v (x, A) (x, (A>B, A>C, A>D) ) v Multilabel v (x, (A, B)) (x, ( (A>C, A>D, B>C, B>D) ) v Label Ranking v (x, (5>4>3>2>1)) (x, ( (5>4, 4>3, 3>2, 2>1) ) y S k h: X S k CS6501 Lecture 3 59

60 Properties of Construction (Zimak et. al 2002, 2003) v Can learn any argmax w i.x function (even when i isn t linearly separable from the union of the others) v Can use any algorithm to find linear separation v Perceptron Algorithm v ultraconservative online algorithm [Crammer, Singer 2001] v Winnow Algorithm v multiclass winnow [ Masterharm 2000 ] v Defines a multiclass margin by binary margin in R KN v multiclass SVM [Crammer, Singer 2001] CS6501 Lecture 3 60

61 This Lecture v Multiclass classification overview v Reducing multiclass to binary v One-against-all & One-vs-one v Error correcting codes v Training a single classifier v Multiclass Perceptron: Kesler s construction v Multiclass SVMs: Crammer&Singer formulation v Multinomial logistic regression CS6501 Lecture 3 61

62 Recall: Margin for binary classifiers v The margin of a hyperplane for a dataset is the distance between the hyperplane and the data point nearest to it Margin with respect to this hyperplane CS6501 Lecture 3 62

63 Multi-class SVM v In a risk minimization framework Ÿ v Goal: D = x, y C F 1. w & V x C > w & Vb x C for all i, y 2. Maximizing the margin CS6501 Lecture 3 63

64 Multiclass Margin CS6501 Lecture 3 64

65 Multiclass SVM (Intuition) v Binary SVM v Maximize margin. Equivalently, Minimize norm of weights such that the closest points to the hyperplane have a score 1 v Multiclass SVM v Each label has a different weight vector (like one-vs-all) v Maximize multiclass margin. Equivalently, Minimize total norm of the weights such that the true label is scored at least 1 more than the second best one CS6501 Lecture 3 65

66 Multiclass SVM in the separable case Recall hard binary SVM Size of the weights. Effectively, regularizer The score for the true label is higher than the score for any other label by 1 CS6501 Lecture 3 66

67 Multiclass SVM: General case Size of the weights. Effectively, regularizer Total slack. Effectively, don t allow too many examples to violate the margin constraint The score for the true label is higher than the score for any other label by 1 Slack variables. Not all examples need to satisfy the margin constraint. Slack variables can only be positive CS6501 Lecture 3 67

68 Multiclass SVM: General case Size of the weights. Effectively, regularizer Total slack. Effectively, don t allow too many examples to violate the margin constraint The score for the true label is higher than the score for any other label by 1 -» i Slack variables. Not all examples need to satisfy the margin constraint. Slack variables can only be positive CS6501 Lecture 3 68

69 Recap: An alternative SVM formulation min w,j,ξ F G w& w + C C ξ C s. t y (w x + b) 1 ξ C ; ξ C 0 i v Rewrite the constraints: ξ C 1 y (w x + b); ξ C 0 v In the optimum, ξ C = max(0, 1 y (w x + b)) v Soft SVM can be rewritten as: min w,j F i G w& w + C C max(0, 1 y (w x + b)) Regularization term Empirical loss CS6501 Lecture 3 69

70 Rewrite it as unconstraint problem Let s define: Δ y, y b δ if y y = 0 if y = y min k F & w G N N w N + C (max C N Δ y C, k + w & N x w & V x ) CS6501 Lecture 3 70

71 Multiclass SVM v Generalizes binary SVM algorithm v If we have only two classes, this reduces to the binary (up to scale) v Comes with similar generalization guarantees as the binary SVM v Can be trained using different optimization methods v Stochastic sub-gradient descent can be generalized CS6501 Lecture 3 71

72 Exercise! v Write down SGD for multiclass SVM v Write down multiclas SVM with Kesler construction CS6501 Lecture 3 72

73 This Lecture v Multiclass classification overview v Reducing multiclass to binary v One-against-all & One-vs-one v Error correcting codes v Training a single classifier v Multiclass Perceptron: Kesler s construction v Multiclass SVMs: Crammer&Singer formulation v Multinomial logistic regression CS6501 Lecture 3 73

74 Recall: (binary) logistic regression min w 1 2 w& w + C log( 1 + e Yq ±(w ² x ± ) ) C Assume labels are generated using the following probability distribution: CS6501 Lecture 3 74

75 (multi-class) log-linear model v Assumption: P y x, w = exp w V & x Vb {F,G, I} exp (w & Vb x) v This is a valid probability assumption. Why? v Another way to write this (with Kesler construction) is P y x, w = exp w & φ(x, y) Vb {F,G, I} exp (w & φ(x, y )) Partition function This often called soft-max CS6501 Lecture 3 75

76 Softmax v Softmax: let s(y) be the score for output y here s(y)=w & φ(x, y) (or w & V x) but it can be computed by other metric. exp s(y) P(y) = exp (s(y)) Vb {F,G, I} v We can control the peakedness of the distribution P(y σ) = exp s y /σ Vb {F,G, I} exp (s(y/σ)) CS6501 Lecture 3 76

77 Example S(dog)=.5; s(cat)=1; s(mouse)=0.3; s(duck)= Dog Cat Mouse Duck score softmax (hard) max (σ 0) softmax (σ=0.5) softmax (σ=2) CS6501 Lecture 3 77

78 Log linear model P y x, w = k º ¹» (k º ¹») ¹ {¼,½, ¾} log P y x, w = log (exp w V & x ) log ( Vb {F,G, I} exp (w & V x)) = w & V x log ( Vb {F,G, I} exp (w & V x)) Linear function Except this term Note: CS6501 Lecture 3 78

79 Maximum log-likelihood estimation v Training can be done by maximum log-likelihood estimation i.e. max k D={(x C, y C )} k º ¹» P(D w = Π C ¹ {¼,½, ¾} log P(D w (k ¹ º» ) log P(D w = C[w & V x C log exp (w & V x C ) Vb {F,G, I} ] CS6501 Lecture 3 79

80 Maximum a posteriori D={(x C, y C )} Can you use Kesler construction to rewrite this formulation? P w D P w P D w max k F G V w & V w V + C C[w & V x C log exp (w & V x C ) Vb {F,G, I} ] or min k F G V w & V w V + C C [ log exp w & V x C w V Vb {F,G, I} & x C ] CS6501 Lecture 3 80

81 Comparisons v v v v Multi-class SVM: F min & w k G N N w N + C C(max (Δ y C, k + w & N N x w & V x)) Log-linear model w/ MAP (multi-class) F N {F,G, I} & x C ] min k w G N & N w N + C C [ log exp w & N x C w V Binary SVM: F min w G w& w + C C max(0, 1 y (w x )) Log-linear mode (logistic regression) min w F G w& w + C log( 1 + e Yq ±(w ² x ± C ) ) CS6501 Lecture 3 81