The Solution Path of the Slab Support Vector Machine

Transcription

1 CCCG 2008, Mntréal, Québec, August 3 5, 2008 The Slutin Path f the Slab Supprt Vectr Machine Michael Eigensatz Jachim Giesen Madhusudan Manjunath Abstract Given a set f pints in a Hilbert space that can be separated frm the rigin. The slab supprt vectr machine (slab SVM) is an ptimizatin prblem that aims at finding a slab (tw parallel hyperplanes whse distance the slab width is essentially fixed) that enclses the pints and is maximally separated frm the rigin. Extreme cases f the slab SVM include the smallest enclsing ball prblem and an interplatin prblem that was used (as the slab SVM itself) in surface recnstructin with radial basis functins. Here we shw that the path f slutins f the slab SVM, i.e., the slutin parametrized by the slab width is piecewise linear. Intrductin Data structures used in fields like graphics, visualizatin and learning ften have many free parameters. In mst cases a gd chice f these parameters is nt bvius. Cmputatinal gemetry was facing similar prblems: fr example when using alpha shapes [Ede95] fr surface recnstructin r in bi-gemetric mdeling the questin arises as t what value t chse fr alpha. Cmputatinal gemetry [Ede95, ELZ02, GCPZ06] gave an answer t this questin that can be seminal als fr the afrementined areas f cmputer science, namely, d nt cmpute the slutin fr a fixed mre r less well chsen value f the parameter, but cmpute the whle spectrum f structures and then lk fr gd slutins in this spectrum. One methd t determine a gd structure is tplgical persistence pineered by Edelsbrunner, Harer and Zmrdian [ELZ02]. Here we investigate an ptimizatin prblem that has its rts in machine learning and was als applied in varius frms t the surface recnstructin prblem. The prblem is called slab supprt vectr machine (slab SVM) [SGS04] and takes as input a set f data pints in a Hilbert space that can be separated frm the rigin and aims at finding a slab (tw parallel hyperplanes whse width is essentially fixed as δ > 0) that enclses the pints and is maximally separated frm the rigin. Applied Gemetry Grup, ETH Zürich, eigensatz@inf.ethz.ch Institut für Infrmatik, Friedrich-Schiller-Universität Jena, giesen@minet.uni-jena.de Max-Planck Institut für Infrmatik, manjun@mpi-inf.mpg.de The slab SVM has fund applicatins in surface recnstructin [SGS04], and quantile estimatin and nvelty detectin [SS02]. In these applicatins the data pints reside in d-dimensinal Euclidean space but are mapped by a feature map int anther (ften infinite dimensinal) Hilbert space. The structure f the slab SVM is such that the feature map des nt have t be given explicitly, but nly implicitly thrugh a psitive kernel: the dual ptimizatin prblem f the slab SVM depends nly n the pairwise inner prducts f the data pints. A psitive kernel can be used t replace these inner prducts withut changing the nature (cnvex quadratic prgram) f the ptimizatin prblem. The parameter we are interested in is δ, which essentially fixes the width f the slab. In the applicatins, it is difficult t tell befrehand what a gd chice f δ is. Hence in the spirit f the cmputatinal gemetry apprach we want t cmpute the slutin t the slab SVM fr all values f δ. Once we have this spectrum f slutins ther methds can be emplyed t find gd chices fr δ. Here we d nt want t discuss hw such methds culd lk like, but fcus n cmputing the slutin spectrum. We shw that the slutin path f the slab SVM, i.e., the slutin parametrized by δ is piecewise linear. Our arguments prvide a cmplete gemetric characterizatin f the turning pints (ndes) f the slutin path. Our results are in spirit similar t results f Hastie et al. [HRTZ04] wh btained the piecewise linearity f the slutin f the classificatin supprt vectr machine [SS02]. Thugh bth results give piecewise linear slutin paths, the parameters are different in nature and s are the means t establish the results. Our prf is f gemetric nature, whereas Hastie et al. use algebraic arguments. 2 The slab SVM Given data pints X = {x,..., x n } H, where H is a Hilbert space with inner prduct,, such that the data pints can be separated frm the rigin by a hyperplane, i.e., there exists w H\{0} and ρ 0 such that w, x i ρ fr all i =,...,n. The distance f the hyperplane {x H : w, x = ρ} t the rigin f H is given as ρ/ w, where the nrm f

2 20th Canadian Cnference n Cmputatinal Gemetry, 2008 w in H is defined as usual by w = w, w. The slab SVM is the fllwing cnvex quadratic ptimizatin prblem that aims at finding the slab (the space between tw parallel hyperplanes) with width δ/ w that cntains all the data pints and minimizes 2 w 2 ρ, i.e., essentially maximizes the distance f the slab t the rigin (see als Figure ): 2 w 2 ρ ρ w, x i ρ + δ fr all i =,...,n and H is the kernel reprducing Hilbert space. Since the dual f the slab SVM nly depends n the inner prducts f the data pints, we can replace x i, x j by k(x i, x j ). A ppular psitive kernel is the Gaussian k(x i, x j ) = exp ( x i x j 2 ) 2σ 2, which is an example f a s called radial basis functin kernel, i.e., a kernel that nly depends n the distance x i x j. The data pints x i = φ(y i ) are linearly independent and the Gram matrix ( k(x i, x j ) ) assciated with the Gaussian kernel is psitive, i.e., it has full rank and thus is invertible. In the fllwing, we always assume that the data pints x i are linear independent. (ρ+δ) / w w. ρ/ w 3 Tw extreme cases The bjective functin f the slab SVM might lk smewhat arbitrary at a first glance. Cnsidering the extreme cases δ = and δ = 0 helps t get a better understanding f the gemetry behind it. Figure : The gemetric set-up fr the slab SVM. Nte that the slab SVM prblem is always feasible since (w, ρ) = (0, 0) is always cntained in the cnstraint plytpe. The Lagrangian dual t this prblem can be derived frm the saddle pint cnditin fr the Lagrangian 3. The pen slab SVM We dente as pen slab SVM the special case δ = f the slab SVM, see Figure 2 fr the gemetric set-up. The pen slab SVM ptimizatin prblem reads as 2 w 2 ρ ρ w, x i fr all i =,...,n L(w, ρ, α, β) = 2 w 2 ρ α i ( w, x i ρ) + i= β i ( w, x i ρ δ), i= where α i, β i 0. The saddle pint cnditin gives L/ w = 0 which implies w = i= (α i β i )x i and L/ ρ = 0 which implies i= (α i β i ) = frm which the dual fllws w. ρ / w min n α,β 2 i,j= (α i β i )(α j β j ) x i, x j + δ i= β i α i, β i 0 fr all i =,...,n. i= (α i β i ) = In mst applicatins [SS02] the data pints are btained frm applying a feature map φ t input data pints y,...,y n R d, i.e., x i = φ(y i ) H, where the feature map is nt given explicitly, but implicitly in frm f a psitive kernel functin k : R d R d R, i.e., x i, x j = φ(y i ), φ(y j ) = k(x i, x j ). Figure 2: The gemetric set-up fr the pen slab SVM. As in the general case, we can derive the Lagrangian dual f the pen slab SVM and get the fllwing ptimizatin prblem in the dual variables α i, i =,...,n, min α 2 i,j= α iα j x i, x j α i 0 fr all i =,...,n. i= α i =

3 CCCG 2008, Mntréal, Québec, August 3 5, 2008 Observe that the ptimal value f this prblem is the distance f the rigin t the cnvex hull f the data pints x,..., x n. The saddle pint cnditin L/ w = 0 implies w = α i x i. i= Hence the ptimal vectr w is the shrtest vectr frm the rigin t the cnvex hull f the data pints and w is the distance f rigin t this cnvex hull. If the data pints x i H are btained frm data pints y i R d that were mapped t H by a feature map implicitly given by a radial basis functin kernel r( ), i.e., by replacing inner prducts y i, y j by r( y i y j ), then the pen slab SVM is equivalent t cmputing the smallest enclsing ball f the data pints x i, see [SS02]. 3.2 The zer slab SVM As zer slab SVM we dente the slab SVM fr the case δ = 0, see Figure 3 fr the gemetric set-up. The zer slab SVM ptimizatin prblem reads as 2 w 2 ρ w, x i = ρ fr all i =,...,n It is wrthwhile t nte that the dual ptimizatin prblem bils dwn t slving a linear system. We derive frm the saddle pint cnditin 0 = L(w, ρ, α) α j = w, x j ρ fr all j =,...,n, which tgether with w = i= α ix i implies α i x i, x j = ρ fr all j =,...,n, i= which is a linear system with unknwn right-hand side ρ fr the dual variables α i. Substituting γ i = α i /ρ gives the linear system γ i x i, x j = fr all j =,...,n, i= which can be slved fr the γ i (assuming the Gram matrix ( x, x j ) has full rank). Frm the γ i we can cmpute ρ as ρ = ( i= γ i), all the α i as α i = ργ i and finally w as w = α i x i = ρ γ i x i. i= i= The linear system fr the γ i was studied extensively in cmputer graphics fr implicit surface recnstructin [CBC + 0]. 4 Surface recnstructin. w ρ / w Figure 3: The gemetric set-up fr the zer slab SVM. The ptimizatin prblem reads in dual variables α i as min α 2 i,j= α iα j x i, x j i= α i = Nte that the ptimal value f this prblem is the distance f the rigin t the affine hull f the data pints x,...,x n. Again we have w = i= α ix i, and thus w is the shrtest vectr frm the rigin t the affine hull f the data pints and w is the distance f rigin t this affine hull. Figure 4: An example surface recnstructin (Max- Planck Head: 2022 pints) using the slab SVM fr a fixed (small) value f δ. Let us briefly recapitulate hw the slab SVM can be used directly fr surface recnstructin [SGS04]. Given are sample pints y,..., y n R 3 frm a smth surface

4 20th Canadian Cnference n Cmputatinal Gemetry, 2008 embedded int R 3. These sample pints are mapped int the feature space assciated with the Gaussian kernel. The recnstructin is given implicitly as f (0), where f : R 3 R is the kernel expansin f(x) = w, φ(x) ρ = (α i β i )exp ( x i x 2 ) 2σ 2 ρ, i= where x R 3, φ( ) is the feature map assciated with the Gaussian kernel, and α and β are the slutins t the dual SVM. Nte that ρ can als be cmputed frm the slutin t the slab SVM (r its dual). See Figure 4 fr an example and als nte that especially in the presence f nise ne prbably des nt want t have an interplating slutin (as ne gets it frm the zer slab SVM and the related methd prpsed in [CBC + 0]), but wuld like t allw small slack in terms f a small value f δ > 0. Nte that the slab SVM wrks the same fr surface recnstructin in dimensins beynd three. Since 2 x i 2 is cnstant, i.e., des nt depend n w r ρ, we can drp it frm the bjective functin. This gives if we set w = w x i and refrmulate the cnstraints in the new variable w accrdingly the fllwing versin f the slab SVM: 2 w 2 0 w, x j x i + x i, x j x i 2 δ fr j i This prblem asks fr the shrtest vectr w in the cnstraint plytpe r equivalently the distance f the cnstraint plytpe t the rigin. Nte that this distance prblem is als always feasible, i.e., the cnstraint plytpe des nt becme empty. T see this bserve that w = x i is always in the plytpe. The gain and lse events can be nicely illustrated fr the distance prblem, see Figure 5. 5 States and events Fr a given value f δ (0, ) let (w, ρ) be the ptimal slutin f the slab SVM. We assciate states with the data pints x i, i =,...,n: () lwer supprting, if w, x i = ρ (2) upper supprting, if w, x i = ρ + δ (3) nn-supprting, if neither lwer- nr upper supprting An event ccurs when while decreasing δ the state f any data pint changes. We distinguish tw types f events: a supprting data pint becmes nnsupprting, r a nn-supprting data pint becmes supprting. We call the first type f event a lse event and the secnd type f event a gain event. 6 The Slutin Path Frm the cnstraints i= (α i β i ) = and α i, β i 0 f the dual f the slab SVM we can cnclude that there exists α i > 0. This in turn allws us t cnclude using the Karuhn-Kuhn-Tucker cnditin α i ( w, xi ρ ) = 0 that fr any δ there always exists a lwer supprting data pint. Fr a given δ, let x i be a lwer supprting data pint. The cntinuus dependence f the cefficient α i n the parameter δ implies that α i > 0 fr sme neighbrhd f U(δ ) (0, ). Hence x i is a lwer supprting data pint fr all δ U(δ ). We use this insight t lcally, i.e., fr δ U(δ ), transfrm the slab SVM int an equivalent distance prblem. Nte that we have ρ = w, x i. Thus we can write the bjective functin f the slab SVM as 2 w 2 ρ = 2 w 2 w, x i = 2 w x i 2 2 x i 2. w w Figure 5: The lwer (nn-mving) cnstraints are shwn by thick slid lines and the upper (mving) cnstraints are shwn by thin slid lines. On the left: when the mving cnstraint hits w this cnstraint becmes binding (gain event) and the slutin is n lnger statinary. On the right: nce the mving cnstraint becmes rthgnal t w we lse the nn-mving cnstraint (lse event). The frmulatin f the slab SVM as a distance prblem allws t make sme bservatins. Lemma The slutin t the slab SVM is unique. Prf. There is always a unique pint in the cnvex cnstraint plytpe f an equivalent distance prblem that realizes the distance f the plytpe t the rigin. Lemma 2 There exists a δ 0 such that fr all δ > δ 0 the slutin t the slab SVM is statinary, i.e. des nt vary with δ. Prf. The prf is via the distance prblem. Let x i be ne f the (lwer) supprting data pints f the pen slab SVM. We use this x i t frmulate the distance

5 CCCG 2008, Mntréal, Québec, August 3 5, 2008 prblem. The slutin f the distance prblem at δ = is finite (we can cnclude this frm the prperties f the pen slab SVM). Cming frm small values f δ the cnstraint plytpes f the distance prblem fr these values f δ sweep the cnstraint plytpe f the distance prblem at δ =. Since the slutin t the latter is finite the sweep needs t hit the pint that realizes this finite distance at sme finite value δ 0 f δ. That is, fr all δ > δ 0 the pint x i is lwer supprting fr the slab SVM and we can cnclude that the slutin f the slab SVM can be derived frm this statinary slutin f the distance prblem as w = w + x i and ρ = w, x i. Lemma 3 Fr all 0 < δ < δ 0 the slab SVM has an upper supprting data pint. Prf. By the prf f Lemma 2 we have that at δ 0, the slab SVM needs t have an upper supprting data pint, because nly the upper cnstraints sweep the cnstraint plytpe f the distance prblem at δ =. Assume there exists 0 < δ < δ 0 such that at δ the slab SVM has n upper supprting data pint. Let be the set f all δ with this prperty and let δ = sup. At δ the slab SVM needs t have an upper supprting data pint. T see this nte that there exists a data pint x j that is upper supprting at δ+ε fr all sufficiently small ε > 0. At δ we can derive a distance prblem that is equivalent t the slab SVM fr sme neighbrhd f δ. The data pint x j needs t be upper supprting als fr this distance prblems at δ + ε fr all sufficiently small ε > 0. The cnstraint hyperplane given by w, x j x i + x i, x j x i 2 = δ () fr the data pint x j has all the cnstraint hyperplanes given by w, x j x i + x i, x j x i 2 = δ + ε (2) n ne side. The latter hyperplanes all cntain a pint that realizes the slutin f the crrespnding distance prblem. By the cntinuity f the distance prblem in δ any sequence in the latter pint set cnverges t the slutin f the distance prblem at δ. Hence this slutin needs t be cntained in the cnstraint hyperplane given by Equatin () and x j is an upper supprting data pint fr bth the distance- and the slab SVM prblem at δ. By ur assumptin there needs t exist sme neighbrhd U f δ such that the distance prblem des nt have an upper supprting data pint fr all δ U (0, δ ). This means that the family f hyperplanes given by Equatin (2) sweeps with ε 0, i.e., at δ, ut f the cnstraint plytpe given by the cnstraints w, x j x i + x i, x j x i 2 = 0. But this can nly happen if the cnstraint plytpe f the distance prblem grws while sweeping the hyperplane given by Equatin (2) frm δ + ε t δ ε, which is a cntradictin. Crllary Fr all 0 < δ < δ 0 the slutin t the slab SVM is nn-statinary. We can cnclude that the slutin path f the slab SVM is piecewise linear (since w the pint that realizes the distance f the cnstraint plytpe t the rigin is a piecewise linear curve parametrized by δ). Therem 4 The slutin path f the slab SVM, i.e., the ptimal cefficients α i and β i (in the dual) and w and ρ (in the primal) are piecewise linear functins f δ. Crllary 2 The ptimal slutin w t the slab SVM is a piecewise linear path that cnnects the pint clsest t the rigin n the cnvex hull (slutin at δ = ) f the data pints with the pint clsest t the rigin n the affine hull (slutin at δ = 0) f the data pints. 7 Cnclusins Therem 4 characterizes the slutin path, but des nt immediately suggest an algrithm t cmpute it. But algrithms fr parametrized cnvex quadratic prgrams (such as the slab SVM) are knwn, see fr example [Rit8]. Anther interesting pen questin is abut the cmplexity f the slutin path, i.e., the number f bends. We cnjecture that this cmplexity can be expnential in the number f data pints. Acknwledgments. Jachim Giesen wants t thank Edgar Rams and Bardia Sadri fr valuable discussins n the slab SVM. References [CBC + 0] Jnathan C. Carr, Richard K. Beatsn, Jn B. Cherrie, Tim J. Mitchell, W. Richard Fright, Bruce C. McCallum, and Tim R. Evans. Recnstructin and representatin f 3d bjects with radial basis functins. In SIGGRAPH, pages 67 76, 200. [Ede95] [ELZ02] Herbert Edelsbrunner. The unin f balls and its dual shape. Discrete & Cmputatinal Gemetry, 3:45 440, 995. Herbert Edelsbrunner, David Letscher, and Afra Zmrdian. Tplgical persistence and simplificatin. Discrete & Cmputatinal Gemetry, 28(4):5 533, [GCPZ06] Jachim Giesen, Frédéric Cazals, Mark Pauly, and Afra Zmrdian. The cnfrmal alpha shape filtratin. The Visual Cmputer, 22(8):53 540, 2006.

6 20th Canadian Cnference n Cmputatinal Gemetry, 2008 [HRTZ04] Trevr Hastie, Saharn Rsset, Rbert Tibshirani, and Ji Zhu. The entire regularizatin path fr the supprt vectr machine. Jurnal f Machine Learning Research, 5:39 45, [Rit8] [SGS04] [SS02] Klaus Ritter. On parametric linear and quadratic prgramming prblems. In Mathematical Prgramming: Prceedings f the Internatinal Cngress n Mathematical Prgramming, pages , 98. Bernhard Schölkpf, Jachim Giesen, and Simn Spalinger. Kernel methds fr implicit surface mdeling. In NIPS, Bernhard Schölkpf and Alex Smla. Learning with Kernels: Supprt Vectr Machines, Regularizatin, Optimizatin and Beynd. MIT Press, Cambridge, MA, 2002.