Learning. Szeged, March Faculty of Mathematics and Computer Science, Babeş Bolyai University, Cluj-Napoca/Kolozsvár

Transcription

1 Faculty of Mathematics and Computer Science, Babeş Bolyai University, Cluj-Napoca/Kolozsvár Machine. Supervised and Szeged, March /41

2 2/41 Contents Machine. Supervised and Machine. Supervised and

3 3/41 to get an overview of kernel methods to get an overview of semi-supervised classification kernel methods: change the kernel obtain a new algorithm semi-supervised : use a large unlabeled dataset semi-supervised kernels: use a supervised algorithm with a semi-supervised kernel to obtain a semi-supervised method no need to develop semi-supervised methods use a supervised algorithm with a good and replaceable semi-supervised kernel Machine. Supervised and

4 4/41 of all times 1. C k-means Apriori 5. EM 6. PageRank ( ) 7. AdaBoost Naive Bayes 10. CART website: icdm//index.shtml article: icdm//10algorithms-08.pdf Machine. Supervised and slides with results: icdm//icdm06-panel.pdf

5 5/41 Machine. Supervised machine (ML): subdomain of AI, modeling the most important brain activities: classification, differentiation, prediction supervised : examples with teaching instructions example: learn to differentiate between relevant and spam s unsupervised : no teaching instructions; group similar points into clusters example: find products similar to a given one semi-supervised : halfway between supervised and unsupervised example: learn to classify handwritten digits based on a few labeled and a large set of unlabeled examples spamham (1200x287x16M jpeg) Machine. Supervised and

6 6/ training data: D train = {(x i, y i ) i = 1,..., N}, x i X, y i Y X is the input space, x i s are vectors Y is the label set, y i s are scalars goal: find ˆf : X Y to approximate f given by D train if Y = 2 binary classification if Y > 2 multi-class classification if f : X Y single-label classification if f : X 2 Y multi-label classification Machine. Supervised and

7 7/41 James Mercer (1909): any continuous symmetric, positive semi-definite kernel function can be expressed as a dot product in a high-dimensional space M. Aizerman, E. Braverman, and L. Rozonoer 1964 Boser, Guyon, and Vapnik 1992 (Support Vector Machine) linear algorithm non-linear algorithm φ: feature mapping kernels: k(x, z) = φ(x), φ(z), φ : X H, X R n i.e. cosine of the angle enclosed by the vectors in a high-dimensional space covers all geometric constructions that can be formulated in terms of angles, lengths and distances any positive definite kernel is a dot product in another space; any mapping φ to a dot product space defines a positive definite kernel by φ(x), φ(z) Machine. Supervised and

8 FEJEZET 6. KERNEL MÓDSZEREK (a) 6.1. we ábra. want Az to XOR solve feladat the XOR-problem bal oldali ábra (shown és a nemlineáris above) vetítés, by a linear mely szeparálhatóvá separator; teszi az adatokat data: (0, jobbra 0), (1, 1), (1, 0), (0, 1) easy to observe: this cannot be done in the initial space tóllets a legtávolabb use the following van. Ez összekapcsolható mapping: φ(x) a = 2.4. [ ] x fejezetben említett regularizáció 1 2 x 2 2 2x1 x 2 módszerével, then the dot ugyanis product: ez a hipersík a legstabilabb a lehetséges hipersíkok közül. módszer φ(x) φ(z) kernelesített = x alkalmazásáig sok évnek kellett eltelnie: a paradigma a modern 1 2z x 1z 1 x 2 z 2 + x2 2z2 2 = (x z) 2 = k(x, z) számítógépek kapacitásának nagymértékű bővülésével lett népszerű. it is called the second order homogeneous polynomial kernel 19. példa. [] Mi újat hoz egy lineáris módszer kernelesítése? A kérdésre az XOR feladattal válaszolunk a 4.4 részből. X 1 X X 1 2 (b) A feladat a 6.1.a ábrán látható négy pont két osztályba való sorolása, ahol a osztályok kódjai az XOR függvény kimenetei. A pontjaink X 1 X X X X Machine. Supervised Data Agraphs and 8/41

9 9/41 General purpose kernels linear (dot product): polynomial: RBF (Gaussian): sigmoid: k(x, z) = x, z k(x, z) = (a x, z + b) c ) x z 2 k(x, z) = exp ( 2σ 2 k(x, z) = tanh(a x, z + r) Machine. Supervised and

10 9/41 General purpose kernels linear (dot product): polynomial: RBF (Gaussian): sigmoid: k(x, z) = x, z k(x, z) = (a x, z + b) c ) x z 2 k(x, z) = exp ( 2σ 2 k(x, z) = tanh(a x, z + r) Machine. Supervised and

11 9/41 General purpose kernels linear (dot product): polynomial: RBF (Gaussian): sigmoid: k(x, z) = x, z k(x, z) = (a x, z + b) c ) x z 2 k(x, z) = exp ( 2σ 2 k(x, z) = tanh(a x, z + r) Machine. Supervised and

12 9/41 General purpose kernels linear (dot product): polynomial: RBF (Gaussian): sigmoid: k(x, z) = x, z k(x, z) = (a x, z + b) c ) x z 2 k(x, z) = exp ( 2σ 2 k(x, z) = tanh(a x, z + r) Machine. Supervised and

13 10/41 Kernel trick Given an algorithm which is formulated in terms of a positive definite kernel k, one can construct an alternative algorithm by replacing k by another positive definite kernel k. Machine. Supervised and

14 11/41 Distances kernels 1. kernel (squared) distance: 2. distance kernel: D (2) = diag(k)1 NN + 1 NN diag(k) 2K 1 2 JD(2) J = 1 2 Jdiag(K)1 NN J 1 2 J1 NN diag(k) J + JKJ = JKJ where J = I 1 N 11 = I 1 N 1 NN is the centering matrix (since 1 NN J = 0) Machine. Supervised and

15 12/41 Outline: learn: calculate the class centroids classify: label an unseen example as belonging to the class with closest centroid + + c x c Machine. Supervised and

16 13/41 Binary classification: centers: m + = {x i y i = +1}, m = {x i y i = 1} c + = 1 m + c = 1 m {i y i =+1} {i y i = 1} Let w := c + c and c := (c + + c )/2; label will be determined by w, x c y = sgn x c, w where b := ( c 2 c + 2) /2 x i x i = sgn ( c +, x c, x + b), Machine. Supervised and

17 14/41 Substituting the corresponding expressions for c + and c we obtain y = sgn 1 where b := 1 2 = sgn 1 m 2 m + {i y i =+1} 1 m + {i y i =+1} {(i,j) y i =y j = 1} x, x i 1 k(x, x i ) 1 m m {i y i = 1} k(x i, x j ) 1 m 2 + and k(, ) is an arbitrary kernel function. {i y i = 1} x, x i + b {(i,j) y i =y j =+1} k(x, x i ) + b, k(x i, x j ) Machine. Supervised and

18 15/41 For multi-class classification: c i = 1 C i x j C i x i The Euclidean distance can be written in terms of the dot product: x z 2 = x x 2x z + z z = k(x, x) 2k(x, z) + k(z, z) Since distances from the centroids are to be considered, we can calculate: x c i 2 = x 1 C i = x x 2 C i = k(x, x) 2 C i x i 2 x j C i x x j + 1 C i 2 x jx k x j C i x j C i,x k C i k(x, x j ) + 1 C i 2 x j C i x j C i,x k C i k(x j, x k ) Machine. Supervised and

19 16/41 k-nearest Neighbors Outline: seemingly no separate phase classify: find the k nearest neighbors of the example in question and label with the majority class Machine. Supervised and

20 17/41 assign the label which is in majority among the k-nearest neighbors f (x) = argmax sim(z, x) δ(c, f (z)) c=1,2,...,k z N k (x),z D train (we use sim(z, x) = 1, z, x) { 1, a = b δ(a, b) = 0, otherwise (Kronecker delta) to determine N k (x) the Euclidean distance is used as before, we can rewrite the Euclidean distance using dot products: x z 2 = x x 2x z + z z = k(x, x) 2k(x, z) + k(z, z) Machine. Supervised and

21 18/41 Support Vector Machines Outline: learn: find a separating hyperplane with maximal margin that separates the positive and negative examples classify: on one side of the hyperplane there are the positive points, on the other side the negative points large-margin separation: the probability of misclassification of an unseen example to be minimized this hyperplane is determined only by the points which lie on the marginal hyperplanes Support Vector Machines (s) setting: binary classification Machine. Supervised and

22 19/41 Hyperplane: w x i + b 1 x i s.t. y i = 1 w x i + b 1 x i s.t. y i = 1 Decision function: f (x) = sgn(w x + b) margin of the separating hyperplane: 2/ w optimization problem: min F (w, b) = 1 2 w 2 subject to y i (w x i + b) 1, i = 1,..., l Machine. Supervised and

23 20/41 Linearly non-separable case introduce slack variables; show the level of violation of linear separability new conditions: w x i + b new optimization problem: min F (w, b, ξ) = 1 2 w 2 + C { 1 ξ i if y i = ξ i if y i = 1 l i=1 s.t. y i (w x i + b) 1 ξ i, ξ i 0, i = 1,..., l ξ i Machine. Supervised and

24 21/41 Outline: clustering method; no supervised information is given goal: cluster data into K sets, such that similar points are grouped in the same set determine K cluster centers Machine. Supervised and

25 objective function: where F = K i=1 j=1 N U ij d(x j, c i ) ci, i = 1,..., K are the cluster centers U is the membership matrix of size K N (K clusters, N points) Uij {0, 1}; U ij = 1 if x j belongs to C i (i-th cluster); otherwise 0 d(, ) is a distance, usually the squared Euclidean: d(x, z) = x z 2 F is minimized if the points are assigned to the closest cluster center, provided they are fixed: { 1, if x U ij = j c i 2 x j c k 2, k i 0, otherwise on the other side, if U is fixed, F is minimized by taking the centers of the groups: 1 N c i = N j=1 U U ik x k ij k=1 Machine. Supervised and 22/41

26 23/41 k-means 1. Initialize the centers. 2. Determine U. 3. Compute cost function F. 4. If converged, then STOP. Else update centers using the new U; GOTO step 2. Machine. Supervised and

27 24/41 Fuzzy c-means: generalization of cluster membership values are [0, 1] objective function: F = K i=1 j=1 N U p ij x j c i 2 such that U ij [0, 1], K i=1 U ij = 1, j, p [1, ) if the centers are fixed, F is minimized by U ij = 1 K k=1 ( x j c i x j c k ) 2 N 1 the values which minimize F for fixed U are Machine. Supervised and c i = 1 N j=1 Up ij N U p ij x j j=1

28 25/41 Kernel and fuzzy c-means are limited to (properly) find pairwise linearly separable clusters to obtain non-linear cluster boundaries we use kernels the only expression one needs to rewrite for the kernel-based variant of the algorithm is the distance of a point to a center: φ(x) 1 s j N k=1 = k(x, x) + 1 s 2 j U jk φ(x k ) 2 N U jk U jl k(x k, x l ) 2 s j k,l=1 where s j is the size of the j-th cluster N k=1 U jk k(x k, x) Machine. Supervised and

29 26/41 Kernel 1. Generate a random U which satisfies the conditions, that is make a random assignment of the points. 2. Compute the cost function F. 3. If converged then STOP. Else update U using the update formula used in (= get new U); GOTO step 2. Machine. Supervised and

30 27/41 (SSL) supervised : D = {(x i, y i ) x i X R d, y i { 1, +1}, i = 1,..., l}; find f : X { 1, +1} which agrees with D semi-supervised : D = {(x i, y i ) i = 1,..., l} {x j j = l + 1,..., l + u}, l u, N = l + u; inductive: find f : X { 1, +1} which agrees with D + use the information of D U transductive: find f : DU { 1, +1} by using D = D L D U Machine. Supervised and

31 27/41 (SSL) supervised : D = {(x i, y i ) x i X R d, y i { 1, +1}, i = 1,..., l}; find f : X { 1, +1} which agrees with D semi-supervised : D = {(x i, y i ) i = 1,..., l} {x j j = l + 1,..., l + u}, l u, N = l + u; inductive: find f : X { 1, +1} which agrees with D + use the information of D U transductive: find f : DU { 1, +1} by using D = D L D U Machine. Supervised and

32 28/41 smoothness assumption: If two points x i and x j in a high density region are close, then so should be the corresponding outputs y i and y j. cluster assumption: If two points are in the same cluster, they are likely to be of the same class. manifold assumption: The high dimensional data lie roughly on a low dimensional manifold. regarding dimensionality; but if manifold = approximation of the high-dimensional region smoothness assumption Machine. Supervised and

33 28/41 smoothness assumption: If two points x i and x j in a high density region are close, then so should be the corresponding outputs y i and y j. cluster assumption: If two points are in the same cluster, they are likely to be of the same class. manifold assumption: The high dimensional data lie roughly on a low dimensional manifold. regarding dimensionality; but if manifold = approximation of the high-dimensional region smoothness assumption Machine. Supervised and

34 28/41 smoothness assumption: If two points x i and x j in a high density region are close, then so should be the corresponding outputs y i and y j. cluster assumption: If two points are in the same cluster, they are likely to be of the same class. manifold assumption: The high dimensional data lie roughly on a low dimensional manifold. regarding dimensionality; but if manifold = approximation of the high-dimensional region smoothness assumption Machine. Supervised and

35 29/41 Generative models Low-density separation Graph-based methods Change of representation Machine. Supervised and

36 29/41 Generative models Low-density separation Graph-based methods Change of representation model class conditional density P(x y) and class priors P(y) and use the Bayes theorem for calculating posteriors, which are used for classification e.g. Naive Bayes + EM 1. Train the classifier on the training examples; determine probabilities P(d i c j ) and P(c j ). 2. Repeat while there is improvement: E-step: Classify unlabeled examples using the trained classifier. M-step: Re-estimate the classifier using the previously classified examples; recalculate P(d i c j ) and P(c j ). Machine. Supervised and

37 29/41 Generative models Low-density separation Graph-based methods Change of representation push away a decision boundary from labeled and unlabeled data; natural choice: use a large-margin classifier e.g. Transductive (T) min 1 2 w 2 s.t. y i (w x i + b) 1, i = 1,..., l y j (w x j + b) 1, y j { 1, +1}, j = l + 1,..., N j = l + 1,..., N Machine. Supervised and

38 29/41 Generative models Low-density separation Graph-based methods Change of representation use the graph of the labeled and unlabeled data, represented by weight matrix W ; use graph Laplacian L W e.g. Label Propagation 1. Compute Y (t + 1) = P Y (t). 2. Reset the labeled data, Y L (t + 1) = Y L (0). 3. t = t + 1 and repeat the above steps until convergence. Machine. Supervised and

39 29/41 Generative models Low-density separation Graph-based methods Change of representation find some structure in the whole data set; use the information provided by the labeled and unlabeled data set to create a new representation 1. Build the new representation new distance, dot-product or kernel of the examples. 2. Use a supervised method for obtaining the decision function employing the new representation obtained in the previous step. e.g. PCA, KPCA, LLE, ISOMAP, etc. Machine. Supervised and

40 30/41 /Bootstrapping Simple self-training Until convergence: 1. Train classifier with the labeled data. 2. Classify unlabeled data with the trained classifier. 3. Add the most confident unlabeled points to the training data. 4. GOTO step 1. Committee-based Until convergence: 1. Train separate classifiers on the same labeled data. 2. Make predictions on the unlabeled data. 3. Add the most agreed points to the training set. 4. GOTO step 1. Final prediction: weighted majority vote among all the learners. Machine. Supervised and

41 31/41 G = (V, W ) undirected weighted graph (usually) V = X L X U (training data) W : V V R (similarity function); W (x i, x j ) = sim(x i, x j ) ( = def W ij ) for example: ( ) sim(x i, x j ) = exp x i x j 2 2 σ 2 sparsification techniques: εnn, graphs graph Laplacian: L = D W, D = diag(w 1) nice properties: e.g. positive semi-definite; has as many 0 eigenvalues as connected components contains; etc. Machine. Supervised and

42 32/41 LP build data graph: W ij = exp ( 1 2σ 2 x i x j 2) compute transition probabilities: D = diag(w 1), P = D 1 W vector of labels: f = [f L f U ] 1. i = 0; Initialize f (i) U. 2. f (i+1) = Pf (i). 3. Clamp the labeled data, f (i+1) L = f L. 4. i = i + 1; Unless convergence go to step 2. Machine. Supervised and

43 33/41 can be rewritten as f i = 1 N j=1 W ij N W ik f k k=1 exact solution (L graph Laplacian) minimizes E(f ) = 1 2 f U = L 1 UU L UL f L N W ij (f i f j ) 2 = f Lf i,j=1 LP is equivalent with searching for a minimum cut in the data graph Machine. Supervised and

44 34/41 Transduction induction fix the other labels and compute the new one we minimize: C + i W iy (f i y) 2 from setting the derivative to zero we obtain: y = i W iy f i j W jy Machine. Supervised and

45 35/41 if data lie on a manifold, graph-based distances are better graph-based distances = shortest path distances calculated using a known algorithm: Floyd Warshall, Dijkstra etc. Floyd Warshall algorithm 1. D ij = weight of the edge; if no edge then 2. for k=1:n for i=1:n for j=1:n if D ij > D ik + D kj D ij = D ik + D kj Machine. Supervised and

46 36/41 and k-means here: semi-supervised = graph-based 1. Initial distance matrix /εnn matrix 2. Compute the all-pair shortest path distance matrix using for example the Floyd Warshall algorithm: D ij = shortest path value among all paths from i to j 3. Perform /k-means using these graph distances. Machine. Supervised and

47 37/41 Neighborhood kernel representing each as the average of its neighbors neighborhood structure influences representation: the kernel: φ nbd (x) = 1 N(x) k nbd (x, z) = x N(x) φ b (x) x N(x),z N(z) k b(x, z ) N(x) N(z) Machine. Supervised and

48 38/41 Bagged cluster kernel idea: reweighting the base kernel values by the probability that the points belong to the same cluster use clustering: random initial cluster centers affects the output of the algorithm BCK 1. Run t times, which results in c j (x i ) cluster assignments, j = 1,..., t, i = 1,..., N, c ( ) {1,..., K}. 2. Construct the bagged kernel in the following way: t j=1 k bag (x, z) = [c j(x) = c j (z)] t The final kernel: k(x, z) = k b (x, z) k bag (x, z) Machine. Supervised and

49 39/41 use + semi-supervised (smoothness) assumption add the following member to the objective function: γ I 1 2 N w ij (f i f j ) 2 = γ I f Lf i,j=1 where L is the graph Laplacian, f is the N 1 vector containing the labels of the points; γ I is a parameter determining how much the unlabeled points influence the result/kernel the optimization problem thus becomes: min w,b s.t. 1 2 w 2 + C l ξ i + γ I f Lf i=1 y i (w x i + b) 1 ξ i Machine. Supervised and

50 40/ it can be shown that it is the same problem which is solved in the original formulation, but now using the kernel 10 ˆk(x, z) = k(x, z) k x ( ) 1 1 2γI I + LK NN Lk z where ˆk(, ) is the new semi-supervised kernel, while k(, ) is the base kernel Machine. Supervised and

51 41/41 Machine. Supervised Thank you! Questions? and