Generalization Bounds in Machine Learning. Presented by: Afshin Rostamizadeh

Transcription

1 Generalization Bounds in Machine Learning Presented by: Afshin Rostamizadeh

2 Outline Introduction to generalization bounds. Examples: VC-bounds Covering Number bounds Rademacher bounds Stability bounds

3 Learning Scenario Goal: given feature vectors labels (y 1,..., y m ) drawn i.i.d. from an (x 1,..., x m ) with unknown distribution D, learn a function (hypothesis) h H that minimizes expected loss (risk) over entire distribution, [ ] R(h) =E (x,y) D l(h(x),y) Where l : R R R is the loss function.

4 Learning Scenario Cannot measure risk directly, approximate using the sample S = ((x 1,y 1 ),..., (x m,y m )) R S (h) = 1 m m i=1 l ( h(x i ),y i ) ) How well does R S approximate R? In other words, how well does the algorithm generalize?

5 Generalization Bound

6 Generalization Bound Generalization (risk) bounds show how well R S (h) bounds R(h) as a function of m = S. R(h) R S (h) + complexity(h)

7 Generalization Bound Generalization (risk) bounds show how well R S (h) bounds R(h) as a function of m = S. R(h) R S (h) + complexity(h) Uniform convergence, holds simultaneously over all h H.

8 Generalization Bound Generalization (risk) bounds show how well R S (h) bounds R(h) as a function of m = S. R(h) R S (h) + complexity(h) Uniform convergence, holds simultaneously over all h H. Complexity (capacity) is a measure of the richness of the hypothesis class.

9 Generalization Bound Generalization (risk) bounds show how well R S (h) bounds R(h) as a function of m = S. R(h) R S (h) + complexity(h) Uniform convergence, holds simultaneously over all h H. Complexity (capacity) is a measure of the richness of the hypothesis class. Explicitly shows the tradeoff between complexity and generalization ability.

10 Outline

11 Outline We will cover four main examples,

12 Outline We will cover four main examples, Hypothesis Complexity Bounds: VC-Dimension Covering Number Rademacher Complexity

13 Outline We will cover four main examples, Hypothesis Complexity Bounds: VC-Dimension Covering Number Rademacher Complexity Algorithm Complexity Bounds: Algorithmic Stability

14 Concentration Bounds Hoeffding s Inequality [ 63] - Let be (X 1,..., X m ) independent bounded ( a i X i b i ) random variables. Then the following is true, Pr ( Y E[Y ] ɛ ) 2 exp ( 2ɛ 2 ) m i=1 (b i a i ) 2 where Y = X X m.

15 Concentration Bounds McDiarmid s Inequality [ 89] - Let f be a real-valued function such that for all i, sup f(x 1,..., x i,..., x m ) f(x 1,..., x i,..., x m ) c i then the following is true, Pr ( f(s) E[f] ɛ ) 2 exp ( 2ɛ 2 m i=1 c i ) with respect to i.i.d. sample S =(x 1,..., x m ).

16 Warmup For finite hypothesis set H and bounded loss function l(, ) M the following is true for all h H simultaneously, ( ) 2mɛ 2 Pr ( R(h) R S (h) ɛ ) 2 H exp M 2

17 Warmup For finite hypothesis set H and bounded loss function l(, ) M the following is true for all h H simultaneously, ( ) 2mɛ 2 Pr ( R(h) R S (h) ɛ ) 2 H exp Setting the rhs equal to, we can say with probability at least δ (1 δ) M 2 the following is true, ln( H ) + ln(2/δ) R(h) R S (h)+m 2m

18 Warmup

19 Warmup Proof: for any fixed hypothesis h, by Hoeffding s inequality, we have Pr ( R(h) R S (h) ɛ ) 2 exp ( 2mɛ 2 M 2 ) where in this case, (a i b i ) M/m.

20 Warmup Proof: for any fixed hypothesis h, by Hoeffding s inequality, we have Pr ( R(h) R S (h) ɛ ) 2 exp ( 2mɛ 2 M 2 ) where in this case, (a i b i ) M/m. Taking the union bound for all hypotheses in H completes the proof.

21 Hypothesis Complexity

22 Hypothesis Complexity Bound can be loose for large values of H.

23 Hypothesis Complexity Bound can be loose for large values of H. In general hypothesis class is not even finite!

24 Hypothesis Complexity Bound can be loose for large values of H. In general hypothesis class is not even finite! We must measure complexity in a more careful way.

25 Symmetrization Main Insight: Define the quantity of interest purely over a finite sample. We can now consider the complexity of a hypothesis class over only a finite sample.

26 Symmetrization Main Insight: Define the quantity of interest purely over a finite sample. Pr [ sup h 2 Pr [ sup h R(h) R S (h) ɛ ] R S (h) R S (h) ɛ 2 We can now consider the complexity of a hypothesis class over only a finite sample. ]

27 VC-dimension

28 VC-dimension Consider binary classification, denote the number of dichotomies realized by H over a sample S as follows, Π H (S) ={(h(x 1 ),..., h(x m )) h H}

29 VC-dimension Consider binary classification, denote the number of dichotomies realized by H over a sample S as follows, Π H (S) ={(h(x 1 ),..., h(x m )) h H} The sample S is shattered by H if, Π H (S) = 2 m.

30 VC-dimension Consider binary classification, denote the number of dichotomies realized by H over a sample S as follows, Π H (S) ={(h(x 1 ),..., h(x m )) h H} The sample S is shattered by H if, Π H (S) = 2 m. The shattering coefficient over m points is the defined as, Π H (m) = max S { Π H(S) : S = m}

31 VC-dimension

32 VC-dimension VC-dimension: the size of the largest set that is shattered by H. d : max{m : S = m, Π H (S) = 2 m }

33 VC-dimension VC-dimension: the size of the largest set that is shattered by H. d : max{m : S = m, Π H (S) = 2 m } Example: halfspaces in R n.

34 VC-dimension VC-dimension: the size of the largest set that is shattered by H. d : max{m : S = m, Π H (S) = 2 m } Example: halfspaces in R n. Can always shatter set of (non-degenerate) n points.

35 VC-dimension VC-dimension: the size of the largest set that is shattered by H. d : max{m : S = m, Π H (S) = 2 m } Example: halfspaces in R n. Can always shatter set of (non-degenerate) n points. No configuration of (n + 1) points can be shattered Case 1: Case 2:

36 SVM Margin Bound For linear hypothesis class h(x) = <w, x> with canonical margin, bound the VC-dim independently of the input dimension, d Λ 2 R 2 where the data is held in a ball of radius R centered at the origin and w Λ. 2 w <w,x>=0

37 VC-bound

38 VC-bound Sauer s Lemma - If VC-dim is finite then, Π H (m) O(m d )

39 VC-bound Sauer s Lemma - If VC-dim is finite then, Π H (m) O(m d ) VC-bound [ 71] - for hypothesis class H with 0/1- loss function and finite VC-dimentsion d, the following is holds with probability at least 1 δ, ( d ln(m) ) R(h) R S (h)+o m + ln(1/δ) Shown to be optimal within a log(m) factor.

40 Covering Number

41 Covering Number Def. The ε-covering number of a function class F with respect to a sample S, denoted N p (F, ɛ,s), is the minimum number of ε-balls needed to cover all h(s) in H(S),

42 Covering Number Def. The ε-covering number of a function class F with respect to a sample S, denoted N p (F, ɛ,s), is the minimum number of ε-balls needed to cover all h(s) in H(S), H(S)

43 Covering Number Def. The ε-covering number of a function class F with respect to a sample S, denoted N p (F, ɛ,s), is the minimum number of ε-balls needed to cover all h(s) in H(S), H(S) ε

44 Covering Number Def. The ε-covering number of a function class F with respect to a sample S, denoted N p (F, ɛ,s), is the minimum number of ε-balls needed to cover all h(s) in H(S), H(S) ε ε

45 Covering Number Def. The ε-covering number of a function class F with respect to a sample S, denoted N p (F, ɛ,s), is the minimum number of ε-balls needed to cover all h(s) in H(S), H(S) ε ε ε

46 Covering Number Def. The ε-covering number of a function class F with respect to a sample S, denoted N p (F, ɛ,s), is the minimum number of ε-balls needed to cover all h(s) in H(S), ε ε ε ε H(S) ε ε ε ε ε ε ε

47 Covering Number Def. The ε-covering number of a function class F with respect to a sample S, denoted N p (F, ɛ,s), is the minimum number of ε-balls needed to cover all h(s) in H(S), ε ε ε ε H(S) ε ε ε ε ε ε ε Intuitively, the number of functions needed to ε-approximate (in p-norm) any hypothesis in H.

48 Covering Number Def. The ε-covering number of a function class F with respect to a sample S, denoted N p (F, ɛ,s), is the minimum number of ε-balls needed to cover all h(s) in H(S), H(S) ε ε ε ε ε Intuitively, the number of functions needed to ε ε-approximate (in p-norm) any hypothesis in H. Also, denote N p (F, ɛ,m) = sup N p (F, ɛ,s). ε ε ε ε S: S =m ε

49 Covering Number

50 Covering Number A natural complexity measure for continuous functions.

51 Covering Number A natural complexity measure for continuous functions. Can be applied during the union bound step. Replace each function in l H with an ε/8 approximation.

52 Covering Number A natural complexity measure for continuous functions. Can be applied during the union bound step. Replace each function in l H with an ε/8 approximation. If loss function is L-Lipschitz, then N p (l F, ɛ,m) N p (F, ɛ/l, m).

53 Covering Number Bound Thm. [Pollard 84] For a Lipschitz loss function bounded by M, the following is holds with probability at least, ( Pr sup R(h) R ) S (h) ɛ h H ( ) mɛ 2 O N 1 (H, ɛ,m)e M 2

54 Rademacher Complexity

55 Rademacher Complexity How well does the hypothesis class fit random noise?

56 Rademacher Complexity How well does the hypothesis class fit random noise? Empirical Rademacher Complexity: R S (H) =E σ (sup h H where σi is chosen uniformly from {1,-1}. 2 m m i=1 σ i h(x i ) )

57 Rademacher Complexity How well does the hypothesis class fit random noise? Empirical Rademacher Complexity: R S (H) =E σ (sup h H where σi is chosen uniformly from {1,-1}. This quantity can be computed from data (assuming we can minimize empirical error). 2 m m i=1 σ i h(x i ) )

58 Rademacher Complexity

59 Rademacher Complexity Rademacher Complexity R m (H) =E S [ RS (H) S = m ]

60 Rademacher Complexity Rademacher Complexity R m (H) =E S [ RS (H) S = m ] McDiarmid s inequality exponentially bounds the difference between R S and R m. ln(1/δ) R m (H) R S (H)+ 2m

61 Rademacher Bound Rademacher Bound: Consider binary classification with 0/1 loss function, then the following is true, Pr sup h H R(h) R S (h) ɛ + R m (h) e 2mɛ2 Again, with probability at least 1 δ, ln(1/δ) R(h) R S (h)+r m (H)+ 2m

62 Rademacher Bound Proof (outline): Apply McDiarmid s inequality to, Φ(S) = sup h which is 1/m-Lipschitz. Main insight: use symmetrization type argument to show, R(h) RS (h) E [ Φ(S) ] R m (H)

63 Rademacher Complexity Important qualities: We can compute R S (H) from data. The empirical Rademacher complexity lower bounds the VC-bound! 2d ln(em/d) R(H) 2M m

64 Algorithmic Stability

65 Algorithmic Stability Complexity measure of a specific algorithm (rather than a hypothesis class).

66 Algorithmic Stability Complexity measure of a specific algorithm (rather than a hypothesis class). The usual union-bound term is not need.

67 Algorithmic Stability Complexity measure of a specific algorithm (rather than a hypothesis class). The usual union-bound term is not need. Intuition: How does the algorithm explore the hypothesis space?

68 Algorithmic Stability Algorithm is called β-stable if, for any two samples S and S that differ in one point, the following holds, β sup x,y l(hs (x),y) l(h S (x),y) where hs and hs are the hypothesis produced by the algorithm for samples S and S respectively.

69 Stability Bound Thm. Let hs be the hypothesis returned by a β- stable algorithm with access to a sample S, then the following bound holds with respect to a nonnegative loss function with upper bound M, Pr S ( R(S) RS (S) ɛ + β ) exp ( 2mɛ 2 (mβ + M) 2 ) Thus, probability at least (1 - δ), over the sample S, ln(1/δ) R(S) R S (S)+β(mβ + M) 2m

70 Stability Bound Proof: Apply McDiarmid s inequality to the random variable, Φ(S) =R(h S ) ˆR S (h S ) Stability allows us to bound the mean, and verify the Lipschitz property.

71 Stability Bound Let S and Si differ in the ith coordinate, show Lipschitz behavior, Φ(S) Φ(S i ) = R(h S ) R S (h S ) R(h S i)+ R S i(h S i) R S i(h S i) R S (h S ) }{{} (1) + R(h S ) R(h S i) }{{} (2) We bound the two terms separately.

72 Stability Bound Part (1): R S i(h S i) R S (h S ) = 1 m j i l(h S i,x j ) l(h S,x j ) + l(h S i,x i) l(h S,x i ) β + M m

73 Stability Bound Part (2): R(h S ) R(h S i) = E x [ l(hs,x) l(h S,x) ] β. So we have, c i = Φ ( S) Φ(S i ) 2β M m

74 Bounding the mean, E S [ Φ(S) ] Stability Bound = E S,x [ l(hs,x) ] E S [ 1 The second equality holds by the i.i.d. assumption. m = E S,x [ l(hs,x) ] E S,x [ 1 = E S,x [ 1 m β i m i i l(h S,x i ) ] [ l(hs,x) l(h S i,x) }{{} β l(h S i,x) ] ] ]

75 Stable Algorithms

76 Stable Algorithms Algorithm must be 1/m-stable for bound to converge.

77 Stable Algorithms Algorithm must be 1/m-stable for bound to converge. Kernel Regularized Algorithms: min h R S (h)+λ h 2 K

78 Stable Algorithms Algorithm must be 1/m-stable for bound to converge. Kernel Regularized Algorithms: min h R S (h)+λ h 2 K Thm. Kernel regularized algorithms with κ- bounded kernel and σ-lipschitz and convex loss function, are stable with coefficient, β σ2 κ 2 λm

79 Stable Algorithms B(f g) =F (f) F (g) (f g) F (g) B 2 K (h S h S )+B 2 K (h S h S ) 2σ mλ sup x S h(x) h = h S h S

80 Proof outline: Stable Algorithms B(f g) =F (f) F (g) (f g) F (g) B 2 K (h S h S )+B 2 K (h S h S ) 2σ mλ sup x S h(x) h = h S h S

81 Proof outline: Stable Algorithms Define the Bregman divergence with respect to RKHS convex function F, B(f g) =F (f) F (g) (f g) F (g) B 2 K (h S h S )+B 2 K (h S h S ) 2σ mλ sup x S h(x) h = h S h S

82 Proof outline: Stable Algorithms Define the Bregman divergence with respect to RKHS convex function F, B(f g) =F (f) F (g) (f g) F (g) First show, where B 2 K (h S h S )+B 2 K (h S h S ) 2σ mλ sup x S h(x) h = h S h S

83 Stable Algorithms B 2 K (h S,h S )= h S h S 2 K 2 h 2 K 2σ mλ sup x S h(x) = 2σ mλ sup K(x, ),h K 2σ x S mλ κ h K

84 Stable Algorithms Proof outline: B 2 K (h S,h S )= h S h S 2 K 2 h 2 K 2σ mλ sup x S h(x) = 2σ mλ sup K(x, ),h K 2σ x S mλ κ h K

85 Stable Algorithms Proof outline: Notice, B 2 (h S,h. K S )= h S h S 2 K 2 h 2 K 2σ mλ sup x S h(x) = 2σ mλ sup K(x, ),h K 2σ x S mλ κ h K

86 Stable Algorithms Proof outline: Notice, B (h 2. K S,h S )= h S h S 2 K Using the kernel reproducing property we see, 2 h 2 K 2σ mλ sup x S h(x) = 2σ mλ sup K(x, ),h K 2σ x S mλ κ h K

87 Stable Algorithms h 2 K σκ mλ l(h S (x),y) l(h S (x),y) σ h(x) κσ h K σ2 κ 2 mλ

88 Stable Algorithms Proof outline: h 2 K σκ mλ l(h S (x),y) l(h S (x),y) σ h(x) κσ h K σ2 κ 2 mλ

89 Stable Algorithms Proof outline: Thus, h 2 K σκ mλ, and finally we see that l(h S (x),y) l(h S (x),y) σ h(x) κσ h K σ2 κ 2 mλ

90 Conclusion Generalization bounds: Show that the empirical error closely estimates the true error. Complexity: Several different notions used to measure size of Hypothesis class. VC-dimension (combinatorial) Stability (algorithm specific) Covering Number (continuous hypothesis) Rademacher Complexity (data specific)

91 Additional Material

92 VC-dimension

93 VC-dimension Consider binary classification, denote the number of dichotomies realized by H over a sample S as follows, Π H (S) ={(h(x 1 ),..., h(x m )) h H}

94 VC-dimension Consider binary classification, denote the number of dichotomies realized by H over a sample S as follows, Π H (S) ={(h(x 1 ),..., h(x m )) h H} The sample S is shattered by H if, Π H (S) = 2 m.

95 VC-dimension Consider binary classification, denote the number of dichotomies realized by H over a sample S as follows, Π H (S) ={(h(x 1 ),..., h(x m )) h H} The sample S is shattered by H if, Π H (S) = 2 m. The shattering coefficient over m points is the defined as, Π H (m) = max S { Π H(S) : S = m}

96 VC-dimension

97 VC-dimension VC-dimension: the size of the largest set that is shattered by H. d : max{m : S = m, Π H (S) = 2 m }

98 VC-dimension VC-dimension: the size of the largest set that is shattered by H. d : max{m : S = m, Π H (S) = 2 m } Example: halfspaces in R n.

99 VC-dimension VC-dimension: the size of the largest set that is shattered by H. d : max{m : S = m, Π H (S) = 2 m } Example: halfspaces in R n. Can always shatter set of (non-degenerate) n points.

100 VC-dimension VC-dimension: the size of the largest set that is shattered by H. d : max{m : S = m, Π H (S) = 2 m } Example: halfspaces in R n. Can always shatter set of (non-degenerate) n points. No configuration of (n + 1) points can be shattered Case 1: Case 2:

101 VC-dimension

102 VC-dimension Sauer s Lemma [ 72]: The shattering coefficient can be bounded by the VC-dimension. Π H (m) d i=0 ( ) m i

103 VC-dimension Sauer s Lemma [ 72]: The shattering coefficient can be bounded by the VC-dimension. Π H (m) d i=0 ( ) m i Furthermore, if d is finite, then for m > d, d i=0 ( ) m i ( ) em d = O(m d ) d

104 VC-bound VC-bound [VC 71] - for hypothesis class H with 0/1-loss function, the following is true, Pr [ sup h H R(h) R ] s (h) ɛ 8Π H (m)e mɛ2 32 Using Sauer s Lemma, the following holds with probability at least, 1 δ 2d ln(em/d) + 2 ln(8/δ) R(h) R S (h)+4 m

105 VC-bound (proof outline)

106 VC-bound (proof outline) Symmetrization - Pr [ sup h 2 Pr [ sup h R(h) R S (h) ɛ ] R S (h) R S (h) ɛ 2 where S is another i.i.d. sample of size m. ]

107 VC-bound (proof outline) Symmetrization - Pr [ sup h 2 Pr [ sup h R(h) R S (h) ɛ ] R S (h) R S (h) ɛ 2 ] where S is another i.i.d. sample of size m. Random Signs - for 2 Pr [ sup h 1 m 4 Pr [ sup h i 1 m ( ) 1h(x i ) y 1 ɛ ] i h(x i ) y i 2 i Pr(σ i = 1) = Pr(σ i = 1) = 1/2 σ i 1 h(xi ) y i ɛ 4 ]

108 VC-bound (proof outline)

109 VC-bound (proof outline) Conditioning - [ 4 Pr sup 1 σ i 1 h m h(xi ) y ɛ i 4 4Π H (m) sup h i Pr [ 1 m i ] x 1,..., x m σ i 1 h(xi ) y i ɛ 4 ] x1,..., x m

110 VC-bound (proof outline) Conditioning - [ 4 Pr sup 1 σ i 1 h m h(xi ) y ɛ i 4 4Π H (m) sup h i Pr [ 1 m Concentration bound (Hoeffding)- i ] x 1,..., x m σ i 1 h(xi ) y i ɛ 4 ] x1,..., x m sup h Pr [ 1 m i σ i 1 h(xi ) y i ɛ 4 x 1,..., x m ] 2e mɛ2 32 Taking the expectation of both sides proves the result.

111 SVM Margin Bound For linear hypothesis class h(x) = <w, x> with canonical margin, bound the VC-dim independently of the input dimension, d Λ 2 R 2 where the data is held in a ball of radius R centered at the origin and w Λ. 2 w <w,x>=0

112 SVM Margin Bound Proof: Choose k points that the hypothesis shatters. Then, upper and lower bound E[ k i=1 y ix i 2 ] with respect to uniform +1/-1 labels chosen for y. E [ Upper bound: By independence of y s, k E [ y i x i y j x j ] = E [ y i x i 2] = i=1 i,j also, y i x i = x i R i y i x i 2] thus, E [ k i=1 y ix i 2] kr 2

113 SVM Margin Bound Lower bound: Since the hypothesis (with canonical margin) shatters the sample i y i(w x i ) k thus, i, y i (w x i ) 1 k i y i(w x i ) w i y ix i Λ i y ix i Combining the bounds, shows k 2 Λ 2 E[ k y i x i 2] kr 2 i=1

114 Covering Number

115 Covering Number Def. The ε-covering number of a function class F with respect to a sample S =(x 1,..., x m ), denoted N p (F, ɛ,s), is the minimum number of points vj needed such that, [ ] m 1/p 1 f F, v j : f(x i ) vj i p ɛ n i=1

116 Covering Number Def. The ε-covering number of a function class F with respect to a sample S =(x 1,..., x m ), denoted N p (F, ɛ,s), is the minimum number of points vj needed such that, [ ] m 1/p 1 f F, v j : f(x i ) vj i p ɛ n i=1 Intuitively, the number of epsilon balls needed to cover the hypothesis space.

117 Covering Number Def. The ε-covering number of a function class F with respect to a sample S =(x 1,..., x m ), denoted N p (F, ɛ,s), is the minimum number of points vj needed such that, [ ] m 1/p 1 f F, v j : f(x i ) vj i p ɛ n i=1 Intuitively, the number of epsilon balls needed to cover the hypothesis space. Also, denote N p (F, ɛ,m) = sup N p (F, ɛ,s). S: S =m

118 Covering Number

119 Covering Number A natural complexity measure for continuous functions.

120 Covering Number A natural complexity measure for continuous functions. Can be applied during the conditioning step. Replace each function in l H with an ε/8 approximation.

121 Covering Number A natural complexity measure for continuous functions. Can be applied during the conditioning step. Replace each function in l H with an ε/8 approximation. If loss function is L-Lipschitz, then N p (l F, ɛ,m) N p (F, ɛ/l, m).

122 Covering Number Bound Thm. [Pollard 84] For L-lipschtiz loss function bounded by M, the following is true, Pr ( sup h H R(h) R S (h) ɛ ) 8N 1 (H, ɛ/(8l),m)exp ( mɛ 2 128M 2 )

123 Covering Number Bound To bound the covering number, notice by Jensen s inequality p < q, N p N q Result of Zhang [ 02], for linear hypothesis classes (h(x) = <w, x>), if x < b, w < a, then log 2 N 2 (H, ɛ,m) a2 b 2 ɛ 2 log 2 (2m + 1)

124 Rademacher Complexity R S (H) =E σ (sup h H 2 m m i=1 σ i h(x i ) )

125 Rademacher Complexity Empirical Rademacher Complexity: R S (H) =E σ (sup h H where again σi is selected uniformly from {1,-1}. 2 m m i=1 σ i h(x i ) )

126 Rademacher Complexity Empirical Rademacher Complexity: R S (H) =E σ (sup h H where again σi is selected uniformly from {1,-1}. This quantity can be computed from data. 2 m m i=1 σ i h(x i ) )

127 Rademacher Complexity Empirical Rademacher Complexity: R S (H) =E σ (sup h H where again σi is selected uniformly from {1,-1}. This quantity can be computed from data. Rademacher Complexity R m (H) =E S [ RS (H) S = m ] 2 m m i=1 σ i h(x i ) )

128 Rademacher Complexity Empirical Rademacher Complexity: R S (H) =E σ (sup h H where again σi is selected uniformly from {1,-1}. This quantity can be computed from data. Rademacher Complexity R m (H) =E S [ RS (H) S = m ] McDiarmid s inequality provides an exponentially bounded difference between R S and R m. 2 m m i=1 σ i h(x i ) )

129 Rademacher Bound Rademacher Bound: Consider binary classification with 0/1 loss function, then the following is true, Pr sup h H R(h) R S (h) ɛ + R m (h) e 2mɛ2 Again, with probability at least 1 δ, ln(1/δ) R(h) R S (h)+r m (H)+ 2m

130 Rademacher Bound Proof (outline): Apply McDiarmid s inequality to, Φ(S) = sup h which is 1/m-Lipschitz. Main insight: use symmetrization type argument to show, R(h) RS (h) E [ Φ(S) ] R m (H)

131 Rademacher Complexity Improvements over VC-bound We can compute R S (H) from data. The empirical Rademacher complexity lower bounds the VC-bound! 2d ln(em/d) R(H) 2M m

132 Rademacher Complexity Proof outline: Use Chernoff bounding method, need to show, [ m exp 2 t R(H) ] Π H (m) exp(t 2 M 2 m/2) then by Sauer s Lemma, m R(H) 2d ln(m) t then choose best t so that, + tm 2 m 2d ln(em/d) R(H) 2M m