Online Learning. Jordan Boyd-Graber. University of Colorado Boulder LECTURE 21. Slides adapted from Mohri

Transcription

1 Online Learning Jordan Boyd-Graber University of Colorado Boulder LECTURE 21 Slides adapted from Mohri Jordan Boyd-Graber Boulder Online Learning 1 of 31

2 Motivation PAC learning: distribution fixed over time (training and test), IID assumption. On-line learning: no distributional assumption. worst-case analysis (adversarial). mixed training and test. Performance measure: mistake model, regret. Jordan Boyd-Graber Boulder Online Learning 2 of 31

3 General Online Setting For t = 1 to T : Get instance x t X Predict ŷ t Y Get true label y t Y Incur loss L(ŷ t, y t ) Classification: Y = {0, 1}, L(y, y ) = y y Regression: Y R, L(y, y ) = (y y) 2 Jordan Boyd-Graber Boulder Online Learning 3 of 31

4 General Online Setting For t = 1 to T : Get instance x t X Predict ŷ t Y Get true label y t Y Incur loss L(ŷ t, y t ) Classification: Y = {0, 1}, L(y, y ) = y y Regression: Y R, L(y, y ) = (y y) 2 Objective: Minimize total loss t L(ŷ t, y t ) Jordan Boyd-Graber Boulder Online Learning 3 of 31

5 Plan Experts Perceptron Algorithm Online Perceptron for Structure Learning Jordan Boyd-Graber Boulder Online Learning 4 of 31

6 Prediction with Expert Advice For t = 1 to T : Get instance x t X and advice a t, i Y, i [1, N] Predict ŷ t Y Get true label y t Y Incur loss L(ŷ t, y t ) Jordan Boyd-Graber Boulder Online Learning 5 of 31

7 Prediction with Expert Advice For t = 1 to T : Get instance x t X and advice a t, i Y, i [1, N] Predict ŷ t Y Get true label y t Y Incur loss L(ŷ t, y t ) Objective: Minimize regret, i.e., difference of total loss vs. best expert Regret(T ) = L(ŷ t, y t ) min L(a t,i, y t ) (1) i t t Jordan Boyd-Graber Boulder Online Learning 5 of 31

8 Mistake Bound Model Define the maximum number of mistakes a learning algorithm L makes to learn a concept c over any set of examples (until it s perfect). M L (c) = max x 1,...,x T mistakes(l, c) (2) For any concept class C, this is the max over concepts c. M L (C) = max c C M L(c) (3) Jordan Boyd-Graber Boulder Online Learning 6 of 31

9 Mistake Bound Model Define the maximum number of mistakes a learning algorithm L makes to learn a concept c over any set of examples (until it s perfect). M L (c) = max x 1,...,x T mistakes(l, c) (2) For any concept class C, this is the max over concepts c. M L (C) = max c C M L(c) (3) In the expert advice case, assumes some expert matches the concept (realizable) Jordan Boyd-Graber Boulder Online Learning 6 of 31

10 Halving Algorithm H 1 H; for t 1... T do Receive x t ; ŷ t Majority(H t, a t, x t ); Receive y t ; if ŷ t y t then H t+1 {a H t : a(x t ) = y t }; return H T +1 Algorithm 1: The Halving Algorithm (Mitchell, 1997) Jordan Boyd-Graber Boulder Online Learning 7 of 31

11 Halving Algorithm Bound (Littlestone, 1998) For a finite hypothesis set M Halving(H) lg H (4) After each mistake, the hypothesis set is reduced by at least by half Jordan Boyd-Graber Boulder Online Learning 8 of 31

12 Halving Algorithm Bound (Littlestone, 1998) For a finite hypothesis set M Halving(H) lg H (4) After each mistake, the hypothesis set is reduced by at least by half Consider the optimal mistake bound opt(h). Then VC(H) opt(h) M Halving(H) lg H (5) For a fully shattered set, form a binary tree of mistakes with height VC(H) Jordan Boyd-Graber Boulder Online Learning 8 of 31

13 Halving Algorithm Bound (Littlestone, 1998) For a finite hypothesis set M Halving(H) lg H (4) After each mistake, the hypothesis set is reduced by at least by half Consider the optimal mistake bound opt(h). Then VC(H) opt(h) M Halving(H) lg H (5) For a fully shattered set, form a binary tree of mistakes with height VC(H) What about non-realizable case? Jordan Boyd-Graber Boulder Online Learning 8 of 31

14 Weighted Majority (Littlestone and Warmuth, 1998) for t 1... N do w 1,i 1; for t 1... T do Receive x t ; ŷ t 1 [ y t,i =1 w t y t,i =0 w t Receive y t ; if ŷ t y t then for t 1... N do if ŷ t y t then w t+1,i βw t,i ; else wt+1,i w t,i ] ; Weights for every expert Classifications in favor of side with higher total weight (y {0, 1}) Experts that are wrong get their weights decreased (β [0, 1]) If you re right, you stay unchanged return w T +1 Jordan Boyd-Graber Boulder Online Learning 9 of 31

15 Weighted Majority (Littlestone and Warmuth, 1998) for t 1... N do w 1,i 1; for t 1... T do Receive x t ; ŷ t 1 [ y t,i =1 w t y t,i =0 w t Receive y t ; if ŷ t y t then for t 1... N do if ŷ t y t then w t+1,i βw t,i ; else wt+1,i w t,i ] ; Weights for every expert Classifications in favor of side with higher total weight (y {0, 1}) Experts that are wrong get their weights decreased (β [0, 1]) If you re right, you stay unchanged return w T +1 Jordan Boyd-Graber Boulder Online Learning 9 of 31

16 Weighted Majority (Littlestone and Warmuth, 1998) for t 1... N do w 1,i 1; for t 1... T do Receive x t ; ŷ t 1 [ y t,i =1 w t y t,i =0 w t Receive y t ; if ŷ t y t then for t 1... N do if ŷ t y t then w t+1,i βw t,i ; else wt+1,i w t,i ] ; Weights for every expert Classifications in favor of side with higher total weight (y {0, 1}) Experts that are wrong get their weights decreased (β [0, 1]) If you re right, you stay unchanged return w T +1 Jordan Boyd-Graber Boulder Online Learning 9 of 31

17 Weighted Majority (Littlestone and Warmuth, 1998) for t 1... N do w 1,i 1; for t 1... T do Receive x t ; ŷ t 1 [ y t,i =1 w t y t,i =0 w t Receive y t ; if ŷ t y t then for t 1... N do if ŷ t y t then w t+1,i βw t,i ; else wt+1,i w t,i ] ; Weights for every expert Classifications in favor of side with higher total weight (y {0, 1}) Experts that are wrong get their weights decreased (β [0, 1]) If you re right, you stay unchanged return w T +1 Jordan Boyd-Graber Boulder Online Learning 9 of 31

18 Weighted Majority Let m t be the number of mistakes made by WM until time t Let m t be the best expert s mistakes until time t m t log N + m t log 1 β log 2 1+β (6) Thus, mistake bound is O(log N) plus the best expert Halving algorithm β = 0 Jordan Boyd-Graber Boulder Online Learning 10 of 31

19 Proof: Potential Function Potential function is the sum of all weights Φ t i w t,i (7) We ll create sandwich of upper and lower bounds Jordan Boyd-Graber Boulder Online Learning 11 of 31

20 Proof: Potential Function Potential function is the sum of all weights Φ t i w t,i (7) We ll create sandwich of upper and lower bounds For any expert i, we have lower bound Φ t w t,i = β mt,i (8) Jordan Boyd-Graber Boulder Online Learning 11 of 31

21 Proof: Potential Function Potential function is the sum of all weights Φ t i w t,i (7) We ll create sandwich of upper and lower bounds For any expert i, we have lower bound Φ t w t,i = β mt,i (8) Weights are nonnegative, so i w t,i w t,i Jordan Boyd-Graber Boulder Online Learning 11 of 31

22 Proof: Potential Function Potential function is the sum of all weights Φ t i w t,i (7) We ll create sandwich of upper and lower bounds For any expert i, we have lower bound Φ t w t,i = β mt,i (8) Each error multiplicatively reduces weight by β Jordan Boyd-Graber Boulder Online Learning 11 of 31

23 Proof: Potential Function (Upper Bound) If an algorithm makes an error at round t Φ t+1 Φ t 2 + βφ t 2 (9) Jordan Boyd-Graber Boulder Online Learning 12 of 31

24 Proof: Potential Function (Upper Bound) If an algorithm makes an error at round t Φ t+1 Φ t 2 + βφ t 2 (9) Half (at most) of the experts by weight were right Jordan Boyd-Graber Boulder Online Learning 12 of 31

25 Proof: Potential Function (Upper Bound) If an algorithm makes an error at round t Φ t+1 Φ t 2 + βφ t 2 (9) Half (at least) of the experts by weight were wrong Jordan Boyd-Graber Boulder Online Learning 12 of 31

26 Proof: Potential Function (Upper Bound) If an algorithm makes an error at round t Φ t+1 Φ t 2 + βφ [ ] t 1 + β 2 = Φ t (9) 2 Jordan Boyd-Graber Boulder Online Learning 12 of 31

27 Proof: Potential Function (Upper Bound) If an algorithm makes an error at round t Φ t+1 Φ t 2 + βφ [ ] t 1 + β 2 = Φ t (9) 2 Initially potential function sums all weights, which start at 1 Φ 1 = N (10) Jordan Boyd-Graber Boulder Online Learning 12 of 31

28 Proof: Potential Function (Upper Bound) If an algorithm makes an error at round t Φ t+1 Φ t 2 + βφ [ ] t 1 + β 2 = Φ t (9) 2 Initially potential function sums all weights, which start at 1 Φ 1 = N (10) After m T mistakes after T rounds [ ] 1 + β mt Φ T N (11) 2 Jordan Boyd-Graber Boulder Online Learning 12 of 31

29 Weighted Majority Proof Put the two inequalities together, using the best expert β m T [ ] 1 + β mt ΦT N (12) 2 Jordan Boyd-Graber Boulder Online Learning 13 of 31

30 Weighted Majority Proof Put the two inequalities together, using the best expert β m T [ ] 1 + β mt ΦT N (12) 2 Take the log of both sides [ ] 1 + β mt log β log N + m T log 2 (13) Jordan Boyd-Graber Boulder Online Learning 13 of 31

31 Weighted Majority Proof Put the two inequalities together, using the best expert β m T [ ] 1 + β mt ΦT N (12) 2 Take the log of both sides [ ] 1 + β mt log β log N + m T log 2 (13) Solve for m T m T log N + m T log 1 β [ ] (14) log 2 1+β Jordan Boyd-Graber Boulder Online Learning 13 of 31

32 Weighted Majority Recap Simple algorithm No harsh assumptions (non-realizable) Depends on best learner Downside: Takes a long time to do well in worst case (but okay in practice) Solution: Randomization Jordan Boyd-Graber Boulder Online Learning 14 of 31

33 Perceptron Algorithm Plan Experts Perceptron Algorithm Online Perceptron for Structure Learning Jordan Boyd-Graber Boulder Online Learning 15 of 31

34 Perceptron Algorithm Perceptron Algorithm Online algorithm for classification Very similar to logistic regression (but 0/1 loss) But what can we prove? Jordan Boyd-Graber Boulder Online Learning 16 of 31

35 Perceptron Algorithm Perceptron Algorithm w 1 0; for t 1... T do Receive x t ; ŷ t sgn( w t x t ); Receive y t ; if ŷ t y t then w t+1 w t + y t x t ; else w t+1 w t ; return w T +1 Algorithm 2: Perceptron Algorithm (Rosenblatt, 1958) Jordan Boyd-Graber Boulder Online Learning 17 of 31

36 Perceptron Algorithm Objective Function Optimizes 1 T Convex but not differentiable max (0, y t ( w x t )) (15) t Jordan Boyd-Graber Boulder Online Learning 18 of 31

37 Perceptron Algorithm Margin and Errors If there s a good margin ρ, you ll converge quickly Jordan Boyd-Graber Boulder Online Learning 19 of 31

38 Perceptron Algorithm Margin and Errors If there s a good margin ρ, you ll converge quickly Whenever you se an error, you move the classifier to get it right Convergence only possible if data are separable Jordan Boyd-Graber Boulder Online Learning 19 of 31

39 Perceptron Algorithm How many errors does Perceptron make? If your data are in a R ball and there is a margin ρ y t( v x t ) v (16) for some v, then the number of mistakes is bounded by R 2 /ρ 2 The places where you make an error are support vectors Convergence can be slow for small margins Jordan Boyd-Graber Boulder Online Learning 20 of 31

40 Online Perceptron for Structure Learning Plan Experts Perceptron Algorithm Online Perceptron for Structure Learning Jordan Boyd-Graber Boulder Online Learning 21 of 31

41 Online Perceptron for Structure Learning Binary to Structure Jordan Boyd-Graber Boulder Online Learning 22 of 31

42 Online Perceptron for Structure Learning Binary to Structure Jordan Boyd-Graber Boulder Online Learning 22 of 31

43 Online Perceptron for Structure Learning Binary to Structure Jordan Boyd-Graber Boulder Online Learning 22 of 31

44 Online Perceptron for Structure Learning Generic Perceptron perceptron is the simplest machine learning algorithm online-learning: one example at a time learning by doing find the best output under the current weights update weights at mistakes Jordan Boyd-Graber Boulder Online Learning 23 of 31

45 Online Perceptron for Structure Learning Structured Perceptron Jordan Boyd-Graber Boulder Online Learning 24 of 31

46 Online Perceptron for Structure Learning Perceptron Algorithm Jordan Boyd-Graber Boulder Online Learning 25 of 31

47 Online Perceptron for Structure Learning POS Example Jordan Boyd-Graber Boulder Online Learning 26 of 31

48 Online Perceptron for Structure Learning What must be true? Finding highest scoring structure must be really fast (you ll do it often) Requires some sort of dynamic programming algorithm For tagging: features must be local to y (but can be global to x) Jordan Boyd-Graber Boulder Online Learning 27 of 31

49 Online Perceptron for Structure Learning Averaging is Good Jordan Boyd-Graber Boulder Online Learning 28 of 31

50 Online Perceptron for Structure Learning Averaging is Good Jordan Boyd-Graber Boulder Online Learning 28 of 31

51 Online Perceptron for Structure Learning Smoothing Must include subset templates for features For example, if you have feature (t 0, w 0, w 1 ), you must also have (t 0, w 0 ); (t 0, w 1 ); (w 0, w 1 ) Jordan Boyd-Graber Boulder Online Learning 29 of 31

52 Online Perceptron for Structure Learning Inexact Search? Sometimes search is too hard So we use beam search instead How to create algorithms that respect this relaxation: track when right answer falls off the beam Jordan Boyd-Graber Boulder Online Learning 30 of 31

53 Online Perceptron for Structure Learning Wrapup Structured prediction: when one label isn t enough Generative models can help with not a lot of data Discriminative models are state of the art Jordan Boyd-Graber Boulder Online Learning 31 of 31

54 Online Perceptron for Structure Learning Wrapup Structured prediction: when one label isn t enough Generative models can help with not a lot of data Discriminative models are state of the art More in Natural Language Processing (at least when I teach it) Jordan Boyd-Graber Boulder Online Learning 31 of 31