Support Vector Clustering

Transcription

1 Support Vector Clustering Asa Ben-Hur, David Horn, Hava T. Siegelmann, Vladimir Vapnik Wittawat Jitkrittum Gatsby tea talk 3 July 25 /

2 Overview Support Vector Clustering Asa Ben-Hur, David Horn, Hava T. Siegelmann, Vladimir Vapnik Journal of Machine Learning Research, 2. Main algorithm based on Support vector domain description David M.J Tax, Robert P.W Duin Pattern Recognition Letters, 999. Goal: Divide {x i } N i= into disjoint groups. Idea: Map x i to φ(x i ) (RKHS). 2 Find the minimal enclosing sphere in RKHS. 3 Sphere in RKHS = non-linear contours in the original space. 4 Interpret the contours as the cluster boundaries. 2/

3 Example (a) (b) (c) (d) (a),..,(d): From high to low Gaussian widths. 3/

4 Support Vector Clustering (SVC) Given {x j } N j=, find the smallest enclosing sphere of radius R. a H (RKHS). min R,a R2 s.t. φ(x j ) a 2 H R2. Soft constraints with slack variables ξ j : min R 2 +C R,a,{ξ j } j j= ξ j s.t. φ(x j ) a 2 H R 2 +ξ j, ξ j. Convex problem. One optimum. 4/

5 Support Vector Clustering (SVC) Given {x j } N j=, find the smallest enclosing sphere of radius R. a H (RKHS). min R,a R2 s.t. φ(x j ) a 2 H R2. Soft constraints with slack variables ξ j : min R 2 +C R,a,{ξ j } j j= ξ j s.t. φ(x j ) a 2 H R 2 +ξ j, ξ j. Convex problem. One optimum. 4/

6 Solving SVC With dual variables {β j } j and {µ j } j, Lagragian is L = R 2 +C ξ j j= ( R 2 +ξ j φ(x j ) a 2 H) β j }{{} j= Setting L L L R =, a =, ξ j = leads to stationarity conditions = N j= β j j= ξ j }{{} 2 a = N j= β jφ(x j ), linear combination of the mapped training points 3 β j = C µ j KKT complementarity conditions (necessary for optimality) ( R 2 +ξ j φ(x j ) a 2 ) H βj = 2 ξ j µ j = µ j. 5/

7 Solving SVC With dual variables {β j } j and {µ j } j, Lagragian is L = R 2 +C ξ j j= ( R 2 +ξ j φ(x j ) a 2 H) β j }{{} j= Setting L L L R =, a =, ξ j = leads to stationarity conditions = N j= β j j= ξ j }{{} 2 a = N j= β jφ(x j ), linear combination of the mapped training points 3 β j = C µ j KKT complementarity conditions (necessary for optimality) ( R 2 +ξ j φ(x j ) a 2 ) H βj = 2 ξ j µ j = µ j. 5/

8 Analysis of Support Vectors (β j > ) A: Constraints φ(x j ) a 2 H R2 +ξ j 2 ξ j,β j,µ j B: Complementarity conditions ( R 2 +ξ j φ(x j ) a 2 ) H βj = 2 ξ j µ j = C: Stationarity conditions = N j= β j 2 a = N j= β jφ(x j ) 3 β j = C µ j Consider < β j < C. C3 µ j >. B2 ξ j =. B φ(x j ) a 2 H = R2. φ(x j ) lies on the sphere surface. Call x j a support vector (SV). Call x i with ξ i > a bounded support vector (BSV). ξ i > means φ(x i ) lies outside the sphere by A. B2 µ j =. C3 β j = C. So, low C limits the influence of a BSV on the sphere. 6/

9 Analysis of Support Vectors (β j > ) A: Constraints φ(x j ) a 2 H R2 +ξ j 2 ξ j,β j,µ j B: Complementarity conditions ( R 2 +ξ j φ(x j ) a 2 ) H βj = 2 ξ j µ j = C: Stationarity conditions = N j= β j 2 a = N j= β jφ(x j ) 3 β j = C µ j Consider < β j < C. C3 µ j >. B2 ξ j =. B φ(x j ) a 2 H = R2. φ(x j ) lies on the sphere surface. Call x j a support vector (SV). Call x i with ξ i > a bounded support vector (BSV). ξ i > means φ(x i ) lies outside the sphere by A. B2 µ j =. C3 β j = C. So, low C limits the influence of a BSV on the sphere. 6/

10 Analysis of Support Vectors (β j > ) A: Constraints φ(x j ) a 2 H R2 +ξ j 2 ξ j,β j,µ j B: Complementarity conditions ( R 2 +ξ j φ(x j ) a 2 ) H βj = 2 ξ j µ j = C: Stationarity conditions = N j= β j 2 a = N j= β jφ(x j ) 3 β j = C µ j Consider < β j < C. C3 µ j >. B2 ξ j =. B φ(x j ) a 2 H = R2. φ(x j ) lies on the sphere surface. Call x j a support vector (SV). Call x i with ξ i > a bounded support vector (BSV). ξ i > means φ(x i ) lies outside the sphere by A. B2 µ j =. C3 β j = C. So, low C limits the influence of a BSV on the sphere. 6/

11 Dual Problem Substituting the stationarity conditions into L gives s.t. max {β j } j β j k(x j,x j ) j= β j =, j= β j C i= j= β i β j k(x i,x j ) µ j dropped. β j = C µ j replaced by β j C. {β j } j used to form a = N j= β jφ(x j ) (sphere center). 7/

12 Sphere Enclosure A point y is inside the sphere if f(y) := φ(y) a H R, where radius R := φ(x i ) a H and x i is a SV i.e., β i < C. f(y) is used for cluster assignment. Easy to compute f(y): f(y) = N k(y,y) 2 β j k(x j,y)+ β i β j k(x i,x j ). j= i= j= Contour in data space: {y φ(y) a H = R}. 8/

13 Cluster Assignment Given two points from different clusters, any path that connects them must exit from the sphere. (b) Define an adjacency matrix A {,} N N : { if for all y on the line segment connecting x i,x j, f(y) R A ij = otherwise Clusters := connected components of the graph induced by A. Implemented by sampling a number of points. BSVs can be treated as outliers, or assigned to closest cluster. 9/

14 Example (a) (b) (c) (d) k(x,y) = exp( q x y 2 ) (a): q =. (b): q = 2. (c): q = 24. (d): q = 48. Increasing q (decreasing width): boundary fits more tightly /

15 Iris Data.5 Iris classification data. 3 classes. 4 dimensions. Project to first two principal components /

16 References I 2/