Presented by. Committee

Transcription

1 Presented by Committee

2

3 Learning from Ambiguous Examples K-Means, PCA Mixture- Models,... Semi- Supervised Clustering,... Semi-Supervised Learning, Transductive- Inference, Multiple-Instance Learning, Co-training,... Neural Nets, Perceptrons, SVM,...

4

5 tiger? tiger? Ambiguity

6 - Each image segment is a point ( to avoid clutter, not all segments are shown ) Asymmetry

7 X x X y (x), Y (X) = ±1 color indicates multiinstance discriminant boundary Y (X) = 1 y (x) = 1

8

9

10

11 D = {(x i, y i ) i = 1,..., m} x R d, y {+1, 1}, (x, y) i.i.d. P f : x X R F (x) = sgn f (x) f (x) = w, x γ (x, y) = yf (x) γ (x, y) = y w, x

12 γ(d) γ (x i, y i ) γ(d), i 1 i m -1 +1

13 err P (f) f(x) err D (f) err P (f) err D (f) + Φ(γ(D)) γ(d)

14 γ(d) w A max γ(d) w,γ(d) s.t. y i w, x i γ(d), i w = 1 B min w w s.t. y i w, x i 1, i Margin constraint defines a halfspace Let s start with a simple example In general

15 Using Max-Margin & Consistency y j Y i X i Classifier labels both segments as tiger segments Only one segment labeled as tiger x X i Y i γ(x, Y ) = Y max x X f(x) x X i f (x)

16 Using Max-Margin & Consistency γ(d) γ (X i, Y i ) = Y i max x X i w, x γ(d), i 1 i m -1 +1

17 Using Max-Margin & Consistency

18 Using Max-Margin & Consistency Unambig min w w s.t. y i w, x i 1, i Ambig min w w s.t. Y i max w, x 1, i x X i Convex feasible region Non-convex feasible region Cookie with convex bites removed

19

20 Quadratic objective with non-linear constraints MI-SVM min w 1 2 w 2 2 s.t. Y i max x X i w, x 1, i 2-norm SVM objective Multi-instance margin constraint Ambiguous training data D = {(X i, Y i ) i = 1,..., m} x R d, x X, Y {+1, 1}, (X, Y ) i.i.d. P

21 1 z(i) X i ) ( Y i (max w, x + b = Y i w, xz(i) + b ) x X i z(i) z(i) x z(i) x z(i)

22

23 with MI-SVM

24 100% 90% 80% 70% EM-DD Citation-KNN SVM linear SVM rbf MI-SVM linear MI-SVM poly MI-SVM rbf 60% 50% 40% Elephant Fox Tiger

25 topic 2 topic 1 topic 3 with MI-SVM topic 4

26 100% 90% 80% 70% 60% 50% TST EM-DD MI-SVM poly MI-SVM linear

27 with MI-SVM

28 100% 90% 80% DD EM-DD MI-Neural Nets MI-LogReg MI-Kernels IAPR MI-SVM rbf 70% 60% MUSK 1 MUSK 2

29

30

31 Linear programming relaxation DPBoost min w 1 w m s.t. w conv i=1 (H(x, Y i ) Q) x X i 1-norm SVM objective w 1 = ( w w d ) Q Convex relaxation of margin constraints Ambiguous training data D = {(X i, Y i ) i = 1,..., m} x R d, x X, Y {+1, 1}, (X, Y ) i.i.d. P

32 Using Max-Margin & Consistency H (x, Y ) H (x, Y ) = { w R d Y w, x 1 } { } w w R d Y i max w, x 1 x X i w x X i H(x, Y i )

33 Using Disjunctive Programming DP min w 1 w m s.t. w H(x, Y i ) Q i=1 x X i Q = { w R d w k 0, k }

34 Convexification DP min w 1 w m m s.t. w conv conv H(x, Y i ) Q i=1 i=1 x X i convex hull hull relaxation

35 Using Disjunctive Programming H i {z R d : A i z b i } z conv i η i 0 H i 1. z = i z i z i R d 2. i η i = 1 Linear constraints! 3. A i z i η i b i

36 Algorithm - Part 1 m d # multi-instances # features O(m m) O(m md) d 1000 repeating structure due to representation of convex hull w 1...

37 Parallel Reductions DP min w 1 w m s.t. w conv i=1 H(x, Y i ) QT t x X i hull relaxation conv (S) T conv (S T ) T Feasible regions T 0 = Q T 1 = T 0 H (x 1, 1) T 2 = T 1 H (x 2, 1) x 1, x 2,...

38 Algorithm - Part 2 m d r # multi-instances # features # reductions w 1 O(mr md)...

39 Ambiguous examples sampled from 2D map Goal was to reconstruct the map Naive Algorithm With true disambiguation DPBoost

40

41

42 Linear programming relaxation LNPBoost min w 1 w m s.t. w conv H(x, Y i ) Q i=1 x X i Ambiguous training data 1-norm SVM objective Improved convex relaxation of margin constraints D = {(X i, Y i ) i = 1,..., m} x R d, x X, Y {+1, 1}, (X, Y ) i.i.d. P

43 Convexification Revisited DP min w 1 w m m conv s.t. w conv H(x, Y i ) Q i=1 i=1 x X i F convex hull hull relaxation F

44 Using Cutting Planes F 0 F 1 F 2... F t F w 0, w 1, w 2,... w t F t lim w t = w F t

45 Using Cutting Planes DP DP min min w 1 w 1 w m s.t. s.t. w conv H(x, Y i ) Q F i=1 x X ii Feasible regions F 0 = Q F 1 = F 0 H F 2 = F 1 H F 3 = F 2 H ( ) α1, 1 β 1 ( ) α2, 1 β 2 ( ) α3, 1 β 3 H ( α1 ), 1 β 1 w, α 1 = β 1

46 Using Cutting Planes Convex approximation Ambiguous margin constraint Intersection H(x, Y i ) x X i F t = conv H (x, Y i ) F t x X i Sequential Convexification

47 Using Cutting Planes LNP max β α, w t β,α,u x N s.t. α u x i (α i ) + v x (x), x {e i, 0} β i=1 N u x i (β i ) + v x (1), x {e i, 0} i=1 u x 0, v x 0, x {e i, 0} α 1 1. α, w = β Farkas Lemma describes valid cuts

48 Using Cutting Planes LNP max β α, w t Cut depth β,α,u x N s.t. α u x i (α i ) + v x (x), x {e i, 0} β i=1 N u x i (β i ) + v x (1), x {e i, 0} i=1 u x 0, v x 0, x {e i, 0} α 1 1. Cut normalization α, w = β Farkas Lemma describes valid cuts Balas LNP Cuts are valid cuts for 0-1 disjunctions

49 Using Cutting Planes LNP max β α, w t Cut depth β,α,u x N s.t. α u x i (α i ) + v x (Y x), x X β i=1 N u x i (β i ) + v x (1), x X i=1 u x 0, v x 0, x X α 1 C LNP. Cut normalization α, w = β Farkas Lemma describes valid cuts Balas LNP Cuts are valid cuts for 0-1 disjunctions Andrews LNP Cuts for general halfspace disjunctions

50 Using Cutting Planes lim w t = w F t β α, w t (X i, Y i )

51 ... Algorithm - Part 1 F t RDP t m d c # multi-instances # features # cuts (X t, Y t ) w 1 O(c d)

52 cut depth cut depth cut number score length of weight vector iteration time features slack variables cuts time

53 Algorithm - Part 2 O(c d) d 1000 m d c # multi-instances # features # cuts w 1...

54 normalized score length of weight vector FS iteration accuracy of projected model classifier accuracy measured on test set FS iteration time

55

56 100% 90% 80% DD EM-DD MI-Neural Nets MI-LogReg MI-Kernels IAPR MI-SVM LNPBoost rbf 70% 60% MUSK 1 MUSK 2

57 2% 0% -2% DD EM-DD MI-Neural Nets MI-LogReg MI-Kernels IAPR MI-SVM LNPBoost rbf -4% -6% MUSK 2 - MUSK 1

58 100% 90% 80% 70% 60% 50% 40% Elephant Fox Tiger EM-DD Citation-KNN SVM linear SVM rbf MI-SVM linear MI-SVM poly MI-SVM rbf LNPBoost linear LNPBoost rbf

59 On average, only seven active features in LNPBoost classifiers 100% 90% 80% 70% 60% 50% TST EM-DD MI-SVM poly MI-SVM linear LNPBoost linear

60 Learning with Labeled & Unlabeled Inputs Label Ambiguity

61 Transductive Inference SVM rbf SDP LNPBoost rbf Area Under ROC Curve

62

63

64 Plug-In Approach

65 Explicit Disambiguation

66 Explicit Disambiguation

67

68

69

70