Introduction to Machine Learning CMU-10701

Transcription

1 Introduction to Machine Learning CMU Learning Theory Barnabás Póczos

2 Learning Theory We have explored many ways of learning from data But How good is our classifier, really? How much data do we need to make it good enough? 2

3 Please ask Questions and give us Feedbacks! 3

4 Review of what we have learned so far 4

5 Notation This is what the learning algorithm produces We will need these definitions, please copy it! 5

6 Big Picture Ultimate goal: Estimation error Approximation error Bayes risk Bayes risk Estimation error Approximation error 6

7 Big Picture Estimation error Approximation error Bayes risk Bayes risk 7

8 Big Picture Estimation error Approximation error Bayes risk Bayes risk 8

9 Big Picture: Illustration of Risks Upper bound Goal of Learning: 9

10 11. Learning Theory 10

11 Outline From Hoeffding s inequality, we have seen that Theorem: These results are useless if N is big, or infinite. (e.g. all possible hyperplanes) Today we will see how to fix this with the Shattering coefficient and VC dimension 11

12 Outline From Hoeffding s inequality, we have seen that Theorem: After this fix, we can say something meaningful about this too: This is what the learning algorithm produces and its true risk 12

13 Hoeffding inequality Theorem: Observation: 13

14 McDiarmid s Bounded Difference Inequality It follows that 14

15 Bounded Difference Condition Our main goal is to bound Lemma: Proof: Let g denote the following function: Observation: ) McDiarmid can be applied for g! 15

16 Bounded Difference Condition Corollary: The VapnikChervonenkis inequality does that with the shatter coefficient (and VC dimension)! 16

17 Concentration and Expected Value 17

18 VapnikChervonenkis inequality Our main goal is to bound We already know: VapnikChervonenkis inequality: Corollary: VapnikChervonenkis theorem: 18

19 Shattering 19

20 How many points can a linear boundary classify exactly in 1D? 2 pts 3 pts There exists placement s.t. all labelings can be classified +?? The answer is 2 20

21 How many points can a linear boundary classify exactly in 2D? + 3 pts 4 pts There exists placement s.t. all labelings can be classified +?? The answer is 3 21

22 How many points can a linear boundary classify exactly in 3D? The answer is tetraeder How many points can a linear boundary classify exactly in ddim? The answer is d+1 22

23 Growth function, Shatter coefficient Definition (=5 in this example) Growth function, Shatter coefficient maximum number of behaviors on n points 23

24 Growth function, Shatter coefficient Definition + Growth function, Shatter coefficient + maximum number of behaviors on n points Example: Half spaces in 2D

25 VCdimension Definition Growth function, Shatter coefficient maximum number of behaviors on n points # behaviors Definition: VCdimension Definition: Shattering Note: 25

26 VCdimension Definition # behaviors 26

27 VCdimension

28 Examples 28

29 VC dim of decision stumps (axis aligned linear separator) in 2d What s the VC dim. of decision stumps in 2d? There is a placement of 3 pts that can be shattered ) VC dim 3 29

30 VC dim of decision stumps (axis aligned linear separator) in 2d What s the VC dim. of decision stumps in 2d? If VC dim = 3, then for all placements of 4 pts, there exists a labeling that can t be shattered 3 collinear + 1 in convex hull of other 3 + quadrilateral

31 VC dim. of axis parallel rectangles in 2d What s the VC dim. of axis parallel rectangles in 2d? There is a placement of 3 pts that can be shattered ) VC dim 3 31

32 VC dim. of axis parallel rectangles in 2d There is a placement of 4 pts that can be shattered ) VC dim 4 32

33 VC dim. of axis parallel rectangles in 2d What s the VC dim. of axis parallel rectangles in 2d? If VC dim = 4, then for all placements of 5 pts, there exists a labeling that can t be shattered 4 collinear pentagon in convex hull 1 in convex hull

34 Sauer s Lemma We already know that [Exponential in n] Sauer s lemma: The VC dimension can be used to upper bound the shattering coefficient. Corollary: [Polynomial in n] 34

35 Proof of Sauer s Lemma Write all different behaviors on a sample (x 1,x 2, x n ) in a matrix:

36 Proof of Sauer s Lemma Shattered subsets of columns: We will prove that Therefore, 36

37 Proof of Sauer s Lemma Shattered subsets of columns: Lemma 1 In this example: =7 Lemma 2 for any binary matrix with no repeated rows. In this example:

38 Proof of Lemma Shattered subsets of columns: In this example: =7 Lemma 1 Proof 38

39 Proof of Lemma 2 Lemma 2 for any binary matrix with no repeated rows. Proof Induction on the number of columns Base case: A has one column. There are three cases: ) 1 1 ) 1 1 )

40 Proof of Lemma 2 Inductive case: A has at least two columns. We have, By induction (less columns)

41 Proof of Lemma 2 because

42 VapnikChervonenkis inequality VapnikChervonenkis inequality: [We don t prove this] From Sauer s lemma: Since Therefore, Estimation error 42

43 Linear (hyperplane) classifiers We already know that Estimation error Estimation error Estimation error 43

44 VapnikChervonenkis Theorem We already know from McDiarmid: VapnikChervonenkis inequality: Corollary: VapnikChervonenkis theorem: [We don t prove them] Hoeffding + Union bound for finite function class: 44

45 PAC Bound for the Estimation VC theorem: Error Inversion: Estimation error 45

46 Structoral Risk Minimization Estimation error Approximation error Bayes risk Ultimate goal: Estimation error Approximation error So far we studied when estimation error! 0, but we also want approximation error! 0 Many different variants penalize too complex models to avoid overfitting 46

47 What you need to know Complexity of the classifier depends on number of points that can be classified exactly Finite case Number of hypothesis Infinite case Shattering coefficient, VC dimension PAC bounds on true error in terms of empirical/training error and complexity of hypothesis space Empirical and Structural Risk Minimization 47

48 Thanks for your attention 48