Hou, Ch. et al. IEEE Transactions on Neural Networks March 2011

Transcription

1 Hou, Ch. et al. IEEE Transactions on Neural Networks March 2011

2 Semi-supervised approach which attempts to incorporate partial information from unlabeled data points

3 Semi-supervised approach which attempts to incorporate partial information from unlabeled data points Normally for SSL: a small portion of data is labeled, the rest is unlabeled

4 Semi-supervised approach which attempts to incorporate partial information from unlabeled data points Normally for SSL: a small portion of data is labeled, the rest is unlabeled In current approach: some negative information about unlabeled data is known, i.e. it is known to which category they don t belong to

5 Examples

6 Examples Human, sport, animal?

7 Examples Human, sport, animal?

8 Examples Human, sport, animal? Mountain, water, building?

9 Examples Human, sport, animal? Mountain, water, building?

10 Examples Human, sport, animal? Mountain, water, building? Instead of: We have:???? 0?

11 Some notation TL true labels, χ (T) = t NL negative labels, χ (N) = n UL unlabeled data, χ (U) = u C number of classes

12 A C Introduce two matrices, Y and A t t Y C n u n a l close to 0 a u close to 1 u

13 A C Introduce two matrices, Y and A t a l a l a l a l a l a l a l a l a l a l a l a l a l a l a l a l a l a l a l a l a l t n Y C n u a l close to 0 a u close to 1 u

14 C A Introduce two matrices, Y and A t a l a l a l a l a l a l a l a l a l a l a l a l a l a l a l a l a l a l a l a l a l Y t n u C n u a l a l a l a l a l a l a l a l a l a l a l a l close to 0 a u close to 1

15 A C Introduce two matrices, Y and A t a l a l a l a l a l a l a l a l a l a l a l a l a l a l a l a l a l a l a l a l a l Y t n u C n u a l a l a u a l a u a l a u a u a l a u a l a l a u a u a l a l a l a u a l a u a u a l close to 0 a u close to 1

16 C A Introduce two matrices, Y and A t a l a l a l a l a l a l a l a l a l a l a l a l a l a l a l a l a l a l a l a l a l a l a l a u a l a u a l a u C n a u a l a u a l a l a u a u Y a l a l a l a u a l a u a u t u a u a u a u a u a u a u a u a u a u a u a u a u a u a u n a l close to 0 a u close to 1 u

17 For each node x i calculate edge weight w ij for its k neighbors

18 For each node x i calculate edge weight w ij for its k neighbors Then normalize

19 Algorithm output is F matrix Classification of x i is argmax j F ij

20 Algorithm output is F matrix Classification of x i is argmax j F ij Iterate, for step m+1, possibility of x i belonging to the j th class

21 F t n u C n C Y A a l a l a l a l a l a l a l t t a l a l a l a l a l a l a l a l a l a l a l a l a l a l n n a l a l a u a l a u a l a u a u a l a u a l a l a u a u a l a l a l a u a l a u a u u u a u a u a u a u a u a u a u a u a u a u a u a u a u a u

22 Iterate, for step m+1, possibility of x i belonging to the j th class I (j) is a diagonal matrix with A 1j, A 2j,..., A (t+n+u)j on its diagonal line

23 Iterate, for step m+1, possibility of x i belonging to the j th class I (j) is a diagonal matrix with A 1j, A 2j,..., A (t+n+u)j on its diagonal line Λ to be a diagonal matrix with d i on diagonal

24 Theorem 1. Update rule: F j 0 = Y j

25 Theorem 1. Update rule: F j 0 = Y j

26 Theorem 1. Update rule: F j 0 = Y j

27 Theorem 1. Update rule: F j 0 = Y j

28 Theorem 1. Update rule: F j 0 = Y j

29

30

31 spectral radius of P 1

32 spectral radius of P 1 I (j) elements 0 < a l, a u < 1

33 spectral radius of P 1 I (j) elements 0 < a l, a u < 1 spectral radius ρ(i (j) P) < 1

34 spectral radius of P 1 I (j) elements 0 < a l, a u < 1 spectral radius ρ(i (j) P) < 1

35 spectral radius of P 1 I (j) elements 0 < a l, a u < 1 spectral radius ρ(i (j) P) < 1

36 spectral radius of P 1 I (j) elements 0 < a l, a u < 1 spectral radius ρ(i (j) P) < 1

37 Theorem 2. is the solution to

38 Theorem 2. is the solution to minimize

39 Theorem 2. is the solution to minimize

40 Name # samples # features # classes Coil (image) x32 20 Umist (face) x28 20 USPS (digit) x16 10 Isolet Newsgroup Corel (image)

41 Algorithms: SVM supervised, uses only TL for training

42 Algorithms: SVM supervised, uses only TL for training LapRLS different forms of loss and smoothness functions, treat NL as UL

43 Algorithms: SVM supervised, uses only TL for training LapRLS different forms of loss and smoothness functions, treat NL as UL GRF doesn t use normalized weights, treat NL as UL

44 Algorithms: SVM supervised, uses only TL for training LapRLS different forms of loss and smoothness functions, treat NL as UL GRF doesn t use normalized weights, treat NL as UL Consistency - normalized weights, treat NL as UL

45 Algorithms: SVM supervised, uses only TL for training LapRLS different forms of loss and smoothness functions, treat NL as UL GRF doesn t use normalized weights, treat NL as UL Consistency - normalized weights, treat NL as UL LNP instead of weights uses local linear approximation coefficients, treat NL as UL

46 NL instances: x i

47 NL instances: x i x i

48 NL instances: x i x i x i

49 NL instances: x i TL instances: x i x i

50 NL instances: x i x i x i The more NL/point is given, the more certainty about the classification

51 Name # NL points # NL/point Coil (image) 10 5 Umist (face) 10 5 USPS (digit) 10 5 Isolet Newsgroup 50 2

52

53 USPS Umist Newsgroup Isolet

54

55

56 Four inputs: σ, k, al, au

57 Four inputs: σ, k, a l, a u τ (0, 1); d the average of all distances

58 Four inputs: σ, k, a l, a u τ (0, 1); d the average of all distances a l and a u by searching the grid {0.01, 0.02,..., 0.09} and {0.90, 0.91,..., 0.99}

59 Four inputs: σ, k, a l, a u τ (0, 1); d the average of all distances a l and a u by searching the grid {0.01, 0.02,..., 0.09} and {0.90, 0.91,..., 0.99} k -?

60 Ex: i for Isolet, τ = 10 (i 1) /10 for i = 1, 2,..., 9 USPS Umist Coil Isolet

61 z is a new point (unlabeled) z s label is defined by weighted labels of its neighbors

62 Time complexity is O((t + n + u) 2 D), where D is the dimensionality of the samples Space complexity is O((t + n + u) 2 )

63 Variable information incorporation of different type labels (NL vs. UL)

64 Variable information incorporation of different type labels (NL vs. UL) Too much of parameter tuning, i.e. σ, k, a l, a u

65 Variable information incorporation of different type labels (NL vs. UL) Too much of parameter tuning, i.e. σ, k, a l, a u Negative labeling defeats the purpose of SSL, i.e. annotating negative labels can be even more time consuming than annotating positive labels

66 Introduction to Semi-Supervised Learning. Xiaojin Zhu, Andrew B. Goldberg Semisupervised Learning Using Negative Labels. Chenping Hou, Feiping Nie, Fei Wang, Changshui Zhang, Yi Wu. IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 22, NO. 3, MARCH Humans Perform Semi-Supervised Classification Too. Xiaojin Zhu, Timothy Rogers, Ruichen Qian, Chuck Kalish. Twenty-Second AAAI Conference on Artificial Intelligence (AAAI-07) Unlabeled data: Now it helps, now it doesn t. Aarti Singh, Robert D. Nowak, Xiaojin Zhu. Advances in Neural Information Processing Systems (NIPS) Manifold Regularization: A Geometric Framework for Learning from Examples. Mikhail Belkin, Partha Niyogi, Vikas Sindhwani. 2004