Information Theory and Feature Selection (Joint Informativeness and Tractability)

Size: px

Start display at page:

Download "Information Theory and Feature Selection (Joint Informativeness and Tractability)"

Irene Josephine Marsh
5 years ago
Views:

1 Information Theory and Feature Selection (Joint Informativeness and Tractability) Leonidas Lefakis Zalando Research Labs 1 / 66

2 Dimensionality Reduction Feature Construction Construction X 1,..., X D f 1 (X 1,..., X D ),..., f k (X 1,..., X D ) 2 / 66

3 Dimensionality Reduction Feature Construction Construction X 1,..., X D f 1 (X 1,..., X D ),..., f k (X 1,..., X D ) Examples Principal Component Analysis (PCA) Linear Discriminant Analysis (LDA) Autoencoders (Neural Networks) 2 / 66

4 Dimensionality Reduction Feature Selection Selection X 1,..., X D X s1,..., X sk Approaches Wrappers Embedded methods Filters 3 / 66

5 Dimensionality Reduction Feature Selection Selection X 1,..., X D X s1,..., X sk Wrappers Features are selected relative to the performance of a specific predictor. Example RFE-SVM. 4 / 66

6 Dimensionality Reduction Feature Selection Selection X 1,..., X D X s1,..., X sk Embedded Methods Features are selected internally while optimizing the predictor. Example Decision Trees. 5 / 66

7 Dimensionality Reduction Feature Selection Selection X 1,..., X D X s1,..., X sk Filters Features are assessed based on some goodness-of-fit function Φ that is classifier agnostic. Example Correlation. 6 / 66

8 Information Theory Entropy H(X ) = p(x) log p(x) x X Joint Entropy H(X, Y ) = x X Conditional Entropy p(x, y) log p(x, y) y Y H(Y X ) = p(x)h(y X = x) x X 7 / 66

9 Information Theory Relative Entropy (Kullback-Leibler Divergence ) Mutual Information I (X ; Y ) = x X D(p q) = x X y Y p(x) log p(x) q(x) ( p(x, y) p(x, y) log p(x)p(y) ) dxdy } {{ } D KL (p(x,y) p(x)p(y)) 8 / 66

10 Mutual Information I (X ; Y ) = x X y Y ( p(x, y) p(x, y) log p(x)p(y) ) dxdy } {{ } D KL (p(x,y) p(x)p(y)) I (X ; Y ) = H(Y ) H(Y X ) Reduction in uncertainty of Y if X is known X Y I (X ; Y ) = 0 Y = f (X ) I (X ; Y ) = H(Y ) 9 / 66

11 Feature Selection Classification We will look into feature selection in the context of classification. Given features {X 1,..., X D } R and a class variable Y, we wish to select a subset S of size K << D such that a predictor f : R K Y trained in this projected subspace generalizes well. 10 / 66

12 Entropy & Prediction Given X Y R D {1... C}, f (X ) = Ŷ We define the error variable { 0 if Ŷ = Y E = 1 if Ŷ Y 11 / 66

13 Entropy & Prediction H(E, Y Ŷ ) (1) = H(Y Ŷ ) + H(E Y, Ŷ ) }{{} =0 (1) = H(E Ŷ ) +H(Y E, Ŷ ) }{{} H(E) H(A, B) = H(A) + H(B A) (1) 12 / 66

14 Entropy & Prediction H(Y Ŷ )= H(E, Y Ŷ ) H(E) + H(Y E, Ŷ ) 1 + P e log ( Y -1) H(E)= H (B(1, P e )) H (B(1, 12 ) ) = 1 H(Y E, Ŷ ) =(1 P e ) H(Y E = 0, Ŷ ) +P }{{} e H(Y E = 1, Ŷ ) }{{} =0 log ( Y -1) 13 / 66

15 Fano s Inequality H(Y Ŷ ) 1 + P e log ( Y -1) (2) = H(Y ) I (Y ; Ŷ ) 1 + P e log ( Y -1) = P e H(Y ) I (Y ; Ŷ ) 1 log ( Y -1) 3 H(Y ) 1 I (X ; Y ) = P e log ( Y -1) I (A; B) = H(A) H(A B) (2) Y X Z = I (Y ; X ) I (Y ; Z) (3) 14 / 66

16 Fano s Inequality H(Y Ŷ ) 1 + P e log ( Y -1) (2) = H(Y ) I (Y ; Ŷ ) 1 + P e log ( Y -1) = P e H(Y ) I (Y ; Ŷ ) 1 log ( Y -1) 3 H(Y ) 1 I (X ; Y ) = P e log ( Y -1) I (A; B) = H(A) H(A B) (2) Y X Z = I (Y ; X ) I (Y ; Z) (3) 14 / 66

17 Data Processing Inequality H(Y Ŷ ) 1 + P e log ( Y -1) (2) = H(Y ) I (Y ; Ŷ ) 1 + P e log ( Y -1) = P e H(Y ) I (Y ; Ŷ ) 1 log ( Y -1) 3 H(Y ) 1 I (X ; Y ) = P e log ( Y -1) I (A; B) = H(A) H(A B) (2) Y X Z = I (Y ; X ) I (Y ; Z) (3) 14 / 66

18 Fano s Inequality H(Y Ŷ ) 1 + P e log ( Y -1) (2) = H(Y ) I (Y ; Ŷ ) 1 + P e log ( Y -1) = P e H(Y ) I (Y ; Ŷ ) 1 log ( Y -1) 3 H(Y ) 1 I (X ; Y ) = P e log ( Y -1) I (A; B) = H(A) H(A B) (2) Y X Z = I (Y ; X ) I (Y ; Z) (3) 14 / 66

19 Objective S = argmax I (X S (1), X S (2),..., X S (K); Y ) S, S =K 15 / 66

20 Objective S = argmax I (X S (1), X S (2),..., X S (K); Y ) S, S =K NP-HARD 15 / 66

21 Feature Selection Naive S = argmax S, S =K K I (X S (k); Y ) k=1 Considers the Relevance of each variable individually Does not consider Redundancy Does not consider Joint Informativeness 16 / 66

22 Feature Selection Two Popular Solutions mrmr : Feature Selection Based on Mutual Information: Criteria of Max-Dependency, Max-Relevance, and Min-Redundancy, H.Peng et al. CMIM : Fast Binary Feature Selection with Conditional Mutual Information, F.Fleuret 17 / 66

23 mrmr Relevance Redundancy mrmr D(S, Y ) = 1 S R(S) = 1 S 2 I (X d ; Y ) X d S X d S X l S I (X d ; X l ) Φ(S, Y ) = D(S, Y ) R(S) 18 / 66

24 Forward Selection S 0 = for k = 1... K do z = 0 for X j F \ S k 1 do S S k 1 X j z Φ(S, Y ) if z > z then s s S S end if end for S i S end for return S K 19 / 66

25 mrmr How do we calculate I ( ; )? Discretize Distributions approximated using Parzen windows Parametric Model ˆp(x) = 1 N N δ(x x (i), h) i=1 δ(dx, h) = 1 2π D exp ( dx T Σ 1 dx 2h 2 ) 20 / 66

26 CMIM (Binary Features) Markov Blanket Markov Blanket of variable A 1 1 Wiki Commons 21 / 66

27 CMIM (Binary Features) Markov Blanket For a set M of variables that form a Markov Blanket of X i we have p (F \ {X i, M}, Y X i, M) = p (F \ {X i, M}, Y M) I (F \ {X i, M}, Y ; X i M) = 0 22 / 66

28 CMIM (Binary Features) Markov Blanket For a set M of variables that form a Markov Blanket of X i we have p (F \ {X i, M}, Y X i, M) = p (F \ {X i, M}, Y M) I (F \ {X i, M}, Y ; X i M) = 0 For Feature Selection Too Strong I (Y ; X i M) = 0 22 / 66

29 CMIM (Binary Features) I (X 1,..., X K ; Y ) = H(Y ) H(Y X 1,..X K ) }{{} intractable 23 / 66

30 CMIM (Binary Features) I (X 1,..., X K ; Y ) = H(Y ) H(Y X 1,..X K ) }{{} intractable I (X i ; Y X j ) = H(Y X j ) H(Y X i, X j ) Distributions over triplets of variables 23 / 66

31 CMIM (Binary Features) I (X 1,..., X K ; Y ) = H(Y ) H(Y X 1,..X K ) }{{} intractable I (X i ; Y X j ) = H(Y X j ) H(Y X i, X j ) S 1 argmax d Î (X d ; Y ) S k argmax{ min Î (Y ; X d X Sl ) } d l<k Distributions over triplets of variables 23 / 66

32 Fast CMIM S k argmax{ min Î (Y ; X d X Sl ) } d l<k }{{} Can only decrease 24 / 66

33 Fast CMIM S k argmax{ min Î (Y ; X d X Sl ) } d l<k }{{} Can only decrease ? 5? 4?? 6?? ? 3????????? ps[n] m[n] ps[n] = min Î (Y ; X d X Sl ) l<m(n) 24 / 66

34 Objective S = argmax I (X S (1), X S (1),..., X S (K); Y ) S, S =K CMIM, mrmr (and others) Compromise Joint Informativeness for Tractability 25 / 66

35 Problem {X 1,..., X D } R, Y {1...C} S = argmax I (X S (1), X S (2),..., X S (K); Y ) S, S =K Calculate I (X ; Y ) using a joint law 26 / 66

36 Information Theory Differential Entropy h(x ) = p(x) log (p(x)) dx X Relative Entropy (Kullback-Leibler Divergence ) ( ) p(x) D(p q) = p(x) log dx q(x) Mutual Information I (X ; Y ) = X Y X ( ) p(x, y) p(x, y) log dxdy p(x)p(y) 27 / 66

37 Addressing Intractability I (X ; Y ) = H(X ) H(X Y ) = H(X ) }{{} intractable Y P(Y = y) H(X Y = y) }{{} intractable 28 / 66

38 Addressing Intractability I (X ; Y ) = H(X ) H(X Y ) = H(X ) }{{} intractable Y Parametric model p X Y P(Y = y) H(X Y = y) }{{} intractable 28 / 66

39 Addressing Intractability I (X ; Y ) = H(X ) H(X Y ) = H(X ) }{{} intractable Y P(Y = y) H(X Y = y) }{{} intractable Parametric model p X Y = N(µ y, Σ y ) 28 / 66

40 Addressing Intractability I (X ; Y ) = H(X ) H(X Y ) = H(X ) }{{} intractable Y P(Y = y) H(X Y = y) }{{} tractable Parametric model p X Y = N(µ y, Σ y ) H(X Y ) = 1 2 log( Σ y ) + n (log 2π + 1) / 66

41 Maximum Entropy Distribution Given E(x) = µ, E ( (x µ)(x µ) T ) = Σ then the multivariate Normal x N( µ, Σ) is the Maximum Entropy Distribution 29 / 66

42 Addressing Intractability I (X ; Y ) = H(X ) H(X Y ) = H(X ) }{{} intractable Y P(Y = y) H(X Y = y) }{{} tractable Parametric model p X Y = N(µ y, Σ y ) H(X Y ) = 1 2 log( Σ y ) + n (log 2π + 1) / 66

43 Addressing Intractability I (X ; Y ) = H(X ) H(X Y ) = H(X ) }{{} intractable Y ( ) H(X ) = H π y N(µ y, Σ y ) y P(Y = y) H(X Y = y) }{{} tractable 31 / 66

44 Addressing Intractability I (X ; Y ) = H(X ) H(X Y ) = H(X ) }{{} intractable Y P(Y = y) H(X Y = y) }{{} tractable ( ) H(X ) = H π y N(µ y, Σ y ) y Entropy of Mixture of Gaussians }{{} No Analytical Solution 31 / 66

45 Addressing Intractability I (X ; Y ) = H(X ) H(X Y ) = H(X ) }{{} intractable Y P(Y = y) H(X Y = y) }{{} tractable ( ) H(X ) = H y π y N(µ y, Σ y ) Entropy of Mixture of Gaussians }{{} No Analytical Solution Upper Bound or Approximate ( ) H y π y N(µ y, Σ y ) 31 / 66

46 A Normal Upper Bound p X = Y π y p X Y 32 / 66

47 A Normal Upper Bound p X = Y π y p X Y p N(µ, Σ ) maxent = H(p X ) H(p ) 32 / 66

48 A Normal Upper Bound p X = Y π y p X Y p N(µ, Σ ) maxent = H(p X ) H(p ) I (X ; Y ) H(p ) Y P Y H(X Y ) 32 / 66

49 A Normal Upper Bound p X = Y π y p X Y p N(µ, Σ ) maxent = H(p X ) H(p ) I (X ; Y ) H(p ) Y P Y H(X Y ) I (X ; Y ) H(Y ) = y p y log p y 32 / 66

50 A Normal Upper Bound p X = Y π y p X Y I (X ; Y ) H(p ) Y P Y H(X Y ) I (X ; Y ) H(Y ) = y p y log p y 33 / 66

51 A Normal Upper Bound p X = Y π y p X Y I (X ; Y ) H(p ) Y P Y H(X Y ) I (X ; Y ) H(Y ) = y p y log p y I (X ; Y ) min ( H(p ), Y P Y (H(X Y ) log(p Y )) ) Y P Y H(X Y ) 33 / 66

52 f f* Disjoint GC 2.2 Entropy Mean difference 34 / 66

53 An approximation p X = Y π y p X Y Under mild assumptions y, H(p ) > H(p X Y =y ) we can use an approximation to I (X ; Y ) Ĩ (X ; Y ) = y min(h(p ), H(X Y ) log p y )p y y H(X Y )p y 35 / 66

54 Feature Selection Criterium Mutual Information Approximation S = argmax (Ĩ (XS (1), X S (2),..., X S (K); Y )) S, S =K 36 / 66

55 Forward Selection S 0 = for k = 1... K do s = 0 for X j F \ S k 1 do S S k 1 X j z Î (S ; Y ) if z > z then s s S S end if end for S i S end for return S K Î (S ; Y ) y min (log( Σ ), log( Σ y ) log p y ) p y y log( Σ y )p y 37 / 66

56 Complexity At iteration k we need to calculate j F \ S k 1 Σ Sk 1 X j 38 / 66

57 Complexity At iteration k we need to calculate j F \ S k 1 Σ Sk 1 X j The cost of calculating each determinant is O(k 3 ) 38 / 66

58 Forward Selection S 0 = for k = 1... K do s = 0 for X j F \ S k 1 do S S k 1 X j z Ĩ (S ; Y ) if z > z then s s S S end if end for S i S end for return S K Overall Complexity O( Y F K 4 ) 39 / 66

59 Forward Selection S 0 = for k = 1... K do s = 0 for X j F \ S k 1 do S S k 1 X j z Ĩ (S ; Y ) if z > z then s s S S end if end for S i S end for return S K Overall Complexity O( Y F K 4 ) However / 66

60 Complexity [ ΣSk 1 Σ j,sk 1 Σ Sk 1 X j = Σ T j,s k 1 σ 2 j ] 40 / 66

61 Complexity [ ΣSk 1 Σ j,sk 1 Σ Sk 1 X j = Σ T j,s k 1 σ 2 j ] We can exploit the matrix determinant lemma (twice) ) Σ + uv T = (1 + v T Σ 1 u Σ To compute each determinant in O(n 2 ) 40 / 66

62 Forward Selection S 0 = for k = 1... K do s = 0 for X j F \ S k 1 do S S k 1 X j z Ĩ (S ; Y ) if z > z then s s S S end if end for S i S end for return S K Overall Complexity O( Y F K 3 ) 41 / 66

63 Forward Selection S 0 = for k = 1... K do s = 0 for X j F \ S k 1 do S S k 1 X j z Ĩ (S ; Y ) if z > z then s s S S end if end for S i S end for return S K Overall Complexity O( Y F K 3 ) Faster? 41 / 66

64 Faster? I (X, Z; Y ) = I (Z; Y ) + I (X ; Y Z) I (S k ; Y ) = I (X j; Y S k 1 ) + I (S k 1 ; Y ) }{{} common 42 / 66

65 Faster? I (X, Z; Y ) = I (Z; Y ) + I (X ; Y Z) I (S k ; Y ) = I (X j; Y S k 1 ) + I (S k 1 ; Y ) }{{} common argmax X j F \S k 1 I (X j ; Y S k 1 ) = argmax (H(X j S k 1 ) H(X j Y, S k 1 )) X j F \S k 1 42 / 66

66 Conditional Entropy H(X j S k 1 ) = H(X j S k 1 = s)µ Sk 1 (s)ds R S k 1 = 1 2 log σ2 j S k (log 2π + 1) 2 σ 2 j S k 1 = σ 2 j Σ T j,s k 1 Σ 1 S k 1 Σ j,sk / 66

67 Updating σ 2 j S k 1 σ 2 j S k 1 = σ 2 j Σ T j,s k 1 Σ 1 S k 1 Σ j,sk 1 Assume X i was chosen at iteration k 1 [ ] ΣSk 2 Σ i,sk 2 Σ Sk 1 = Σ T i,s k 2 σ 2 i [ ΣSk 2 0 = k 2 0 T n 2 σi 2 ] + e n+1 Σ T i,s k 2 + Σ i,sk 2 e T n+1 44 / 66

68 Updating Σ 1 S k 1 From Sherman-Morrison formula (Σ + uv T ) 1 = Σ 1 Σ 1 uv T Σ v T Σ 1 u 45 / 66

69 Updating Σ 1 S k 1 From Sherman-Morrison formula (Σ + uv T ) 1 = Σ 1 Σ 1 uv T Σ v T Σ 1 u Σ 1 S n 1 = [ Σ 1 S n 2 1 u βσi 2 1 βσ 2 i u T 1 βσ 2 i ] + 1 βσ 2 i [ u 0 ] [ u T 0 ] 45 / 66

70 Updating Σ 1 S k 1 Σ 1 S n 1 = [ Σ 1 S n 2 1 u βσi 2 1 βσ 2 i u T 1 βσ 2 i ] + 1 βσ 2 i [ u 0 ] [ u T 0 ] σ 2 j S n 1 = σ 2 j + σ2 ji βσ 2 i 1 βσ 2 i Previous Round {}}{ Σ T j,s n 2 Σ 1 S n 2 Σ j,sn 2 O(n 2 ) ] βσi 2 1 βσi 2 u T Σ j,sn 2 Σ T j,s n 1 [ 1 ( Σ T j,s n 1 [ u 0 u σ 2 ji ]) ([ u T 0 ] Σ jsn 1 ) O(n) O(n) 46 / 66

71 Faster! for k = 1... K do s = 0 for X j F \ S k 1 do S S k 1 X j z Î (S ) if z > z then s s S S end if end for S i S end for return S K Overall Complexity O( Y F K 2 ) 47 / 66

72 Faster! for k = 1... K do s = 0 for X j F \ S k 1 do S S k 1 X j z Î (S ) if z > z then s s S S end if end for S i S end for return S K Overall Complexity O( Y F K 2 ) Even Faster? 47 / 66

73 Even Faster? Main bottleneck O(k) σ 2 j S k 1 = σ 2 j Σ T js k 1 Σ 1 S k 1 Σ jsk 1 48 / 66

74 Even Faster? Main bottleneck O(k) σ 2 j S k 1 = σ 2 j Σ T js k 1 Σ 1 S k 1 Σ jsk 1 Skip non-promising features 48 / 66

75 Forward Selection S 0 = for k = 1... K do s = 0 for X j F \ S k 1 do S S k 1 X j z Î (S ) if z > z then s s S S end if end for S i S end for return S K Cheap (O(1)) score c: if c < z then z < z 49 / 66

76 Forward Selection... z = 0 for X j F \ S k 1 do S S k 1 X j c? if c > z then z Î (S ) if z > z then s s S S end if end if end for... Cheap (O(1)) score c: if c < z then z < z 49 / 66

77 An O(1) Bound σ 2 j S k 1 = σ 2 j Σ T js k 1 Σ 1 S k 1 Σ jsk 1 Σ T js k 1 Σ 1 S k 1 Σ jsk 1 =Σ T js k 1 UΛU T Σ jsk 1 Σ T js k max λ i Σ T js i n 1 Σ 1 S k 1 Σ jsk 1 Σ T js k min λ i }{{ i } O(1) 50 / 66

78 An O(1) Bound σ 2 j S k 1 = σ 2 j Σ T js k 1 Σ 1 S k 1 Σ jsk 1 Σ T js k 1 Σ 1 S k 1 Σ jsk 1 =Σ T js k 1 UΛU T Σ jsk 1 Σ T js k max λ i Σ T js i n 1 Σ 1 S k 1 Σ jsk 1 Σ T js k min λ i }{{ i } O(1) 50 / 66

79 An O(1) Bound σ 2 j S k 1 = σ 2 j Σ T js k 1 Σ 1 S k 1 Σ jsk 1 Σ T js k 1 Σ 1 S k 1 Σ jsk 1 =Σ T js k 1 UΛU T Σ jsk 1 Σ T js k max λ i Σ T js i n 1 Σ 1 S k 1 Σ jsk 1 Σ T js k min λ i }{{ i } O(1) 50 / 66

80 An O(1) Bound σ 2 j S k 1 = σ 2 j Σ T js k 1 Σ 1 S k 1 Σ jsk 1 Σ T js k 1 Σ 1 S k 1 Σ jsk 1 =Σ T js k 1 UΛU T Σ jsk 1 Σ T js k max λ i Σ T js i n 1 Σ 1 S k 1 Σ jsk 1 Σ T js k min λ i }{{ i } O(1) 50 / 66

81 An O(1) Bound σ 2 j S k 1 = σ 2 j Σ T js k 1 Σ 1 S k 1 Σ jsk 1 Σ T js k 1 Σ 1 S k 1 Σ jsk 1 =Σ T js k 1 UΛU T Σ jsk 1 Σ T js k max λ i Σ T js i n 1 Σ 1 S k 1 Σ jsk 1 Σ T js k min λ i }{{ i } O(1) 50 / 66

82 EigenSystem Update Problem Given U n, Λ n such that Σ n = U nt Λ n U n 51 / 66

83 EigenSystem Update Problem Given U n, Λ n such that Find U n+1, Λ n+1 Σ n+1 = assume Σ n+1 S n+1 ++ Σ n = U nt Λ n U n [ Σ n v v T 1 ] = U n+1t Λ n+1 U n+1 51 / 66

84 EigenSystem Update U n = [ u n 1 u n 2... u n n λ n 1 0 ], Λ n = λ n n 52 / 66

85 EigenSystem Update U n = [ u n 1 u n 2... u n n λ n 1 0 ], Λ n = λ n n [ Σ n 0 0 T 1 ] [ u n i 0 ] = λ n i [ ] u n i 0 }{{} u n i 52 / 66

86 EigenSystem Update U n = [ u n 1 u n 2... u n n λ n 1 0 ], Λ n = λ n n [ Σ n 0 0 T 1 [ Σ n 0 0 T 1 ] [ u n i 0 ] [ 0 1 ] [ ] = λ n u n i i 0 }{{} u n i ] = [ 0 1 ] 52 / 66

87 EigenSystem Update [ ] Σ n 0 0 T = 1 }{{} Σ u n 1 T. e n+1t λ n } 0 {{ 1 } Λ [ u n 1... e n+1 ] }{{} U Σ n+1 = Σ + e n+1 [ v 0 ] T [ v + 0 ] e T n+1 ( Σ + e n+1 [ v 0 ] T [ v + 0 ] ) e T n+1 u n+1 = λ n+1 u n+1 53 / 66

88 EigenSystem Update U T ( ( Σ + e n+1 [ v 0 Σ + e n+1 [ v 0 ] T [ v + 0 ] T [ v + 0 ] ) e T n+1 u n+1 = λ n+1 u n+1 ] ) e T n+1 U U T }{{} u n+1 = λ n+1 U T u n+1 U U T =I ( ) ((4)) = Λ + e n+1 q T + qe T n+1 U T u n+1 = λ n+1 U T u n+1 U T Σ U = Λ (4) 54 / 66

89 EigenSystem Update ( ) Λ + e n+1 q T + qe T n+1 U T u n+1 = λ n+1 U T u n+1 }{{} Σ Σ n+1 and Σ share eigenvalues. U n+1 = U U 55 / 66

90 EigenSystem Update Σ λi = j (λ j λ) + i<n+1 q 2 i j i,j<n+1 (λ j λ) f (λ) = λ n+1 λ + i q 2 i (λ i λ). 56 / 66

91 EigenSystem Update Σ λi = j (λ j λ) + i<n+1 q 2 i j i,j<n+1 (λ j λ) f (λ) = λ n+1 λ + i q 2 i (λ i λ). i, lim λ λ > i f (λ) = +, lim λ λ < i f (λ) = f (λ) λ = 1 + i q 2 i (λ i λ) / 66

92 57 / 66

93 30 25 Lapack Update 20 CPU time in secs Matrix size Comparison between scratch and update 58 / 66

94 x max λ update λ Lapack /λ Lapack Matrix size Numerical stability (eigenvalues) 59 / 66

95 Set to go... Now that all the machinery is in place, How did we do? 60 / 66

96 Nice Results CIFAR STL INRIA SVMLin Fisher FCBF MRMR SBMLR ttest InfoGain Spec. Clus RelieFF CFS CMTF BAHSIC GC.E GC. MI GKL.E GKL. MI GC.MI was the fastest of the more complex algorithms 61 / 66

97 Influence of Sample Size We use estimates ˆΣ N = 1 N PT P For (sub)-gaussian data we have 2 If N C(t/ɛ) 2 d then ˆΣ N Σ ɛ 2 with probability at least 1 2e ct2 62 / 66

98 Influence of Sample Size We use estimates ˆΣ N = 1 N PT P For (sub)-gaussian data we have 2 If N C(t/ɛ) 2 d then ˆΣ N Σ ɛ However the faster implementations use ˆΣ 1 N 2 with probability at least 1 2e ct2 62 / 66

99 Influence of Sample Size κ(σ) = Σ 1 e Σ 1 b e b = Σ 1 Σ, for d = 2048 and various values of N 63 / 66

100 50 45 Test Accuracy features 25 features 50 features Number of Samples per Class Effect of sample size on performance when using the Gaussian Approximation for the CIFAR dataset. 64 / 66

101 Jointly Informative Feature Selection Made Tractable by Gaussian Modeling L.Lefakis and F.Fleuret Journal of Machine Learning Research, / 66

102 The End Thank You! 66 / 66

Jointly Informative Feature Selection Made Tractable by Gaussian Modeling

Journal of Machine Learning Research 17 216) 1-39 Submitted 1/15; Revised 11/15; Published 1/16 Jointly Informative Feature Selection Made Tractable by Gaussian Modeling Leonidas Lefakis Zalando Research