Kernel Methods for Implicit Surface Modeling

Transcription

1 Max Planck Insttut für blgsche Kybernetk Max Planck Insttute fr Blgcal Cybernetcs Techncal Reprt N. TR-125 Kernel Methds fr Implct Surface Mdelng Bernhard Schölkpf, Jachm Gesen +, Smn Spalnger + June 2004 Max Planck Insttute fr Blgcal Cybernetcs, Tübngen, Germany, emal: bernhard.schelkpf@tuebngen.mpg.de + Department f Cmputer Scence, ETH Zürch, Swtzerland, emal: gesen@nf.ethz.ch,spsmn@nf.ethz.ch Ths reprt s avalable n PDF frmat va annymus ftp at ftp://ftp.kyb.tuebngen.mpg.de/pub/mp-mems/pdf/tr125.pdf. The cmplete seres f Techncal Reprts s dcumented at:

2 Kernel Methds fr Implct Surface Mdelng Bernhard Schölkpf y, Jachm Gesen +Λ & Smn Spalnger + y Max Planck Insttute fr Blgcal Cybernetcs, Tübngen, Germany bernhard.schelkpf@tuebngen.mpg.de + Department f Cmputer Scence, ETH Zürch, Swtzerland gesen@nf.ethz.ch,spsmn@nf.ethz.ch Abstract We descrbe methds fr cmputng an mplct mdel f a hypersurface that s gven nly by a fnte samplng. The methds wrk by mappng the sample pnts nt a reprducng kernel Hlbert space and then determnng regns n terms f hyperplanes. 1 Intrductn Suppse we are gven a fnte samplng (n machne learnng terms, tranng data) x 1 ;:::;x m 2; where the dman s sme hypersurface n Eucldean space R d. The case d = 3 s especally nterestng snce these days there are many devces, e.g., laser range scanners, that allw the acqustn f pnt data frm the bundary surfaces f slds. Fr further prcessng t s ften necessary t transfrm ths data nt a cntnuus mdel. Tday the mst ppular apprach s t add cnnectvty nfrmatn t the data by transfrmng them nt a trangle mesh (see [4] fr an example f such a transfrmatn algrthm). But recently als mplct mdels, where the surface s mdeled as the zer set f sme suffcently smth functn, ganed sme ppularty [1]. One advantage f mplct mdels s that they easly allw the dervatn f hgher rder dfferental quanttes such as curvatures. Anther advantage s that an nsde-utsde test,.e., testng whether a query pnt les n the bunded r unbunded sde f the surface, bls dwn t determnng the sgn f a functn-evaluatn at the query pnt. Insde-utsde tests are mprtant when ne wants t ntersect tw slds. The gal f ths paper s, lsely speakng, t fnd a functn whch takes the value zer n a surface whch (1) cntans the tranng data and (2) s a reasnable mplct mdel f. T capture prpertes f ts shape even n the abve general case, we need t explt sme structure n. In lne wth a szeable amunt f recent wrk n kernel methds [12], we assume that ths structure s gven by a (pstve defnte) kernel,.e., a real valued functn k n whch can be expressed as k(x; x 0 )=hφ(x); Φ(x 0 ) (1) Λ Partally supprted by the Swss Natnal Scence Fundatn under the prject Nn-lnear manfld learnng.

3 w. ρ/ w Φ( x) ξ/ w Fgure 1: In the 2-D ty example depcted, the hyperplane hw; Φ(x) = ρ separates all but ne f the pnts frm the rgn. The utler Φ(x) s asscated wth a slack varable ο, whch s penalzed n the bjectve functn (4). The dstance frm the utler t the hyperplane s ο=kwk; the dstance between hyperplane and rgn s ρ=kwk. The latter mples that a small kwk crrespnds t a large margn f separatn frm the rgn. fr sme map Φ nt a Hlbert space H. The space H s the reprducng kernel Hlbert space (RKHS) asscated wth k, and Φ s called ts feature map. A ppular example, n the case where s a nrmed space, s the Gaussan (where ff>0) k(x; x 0 )=exp kx x0 k 2 : (2) 2 ff 2 The advantage f usng a pstve defnte kernel as a smlarty measure s that t allws us t cnstruct gemetrc algrthms n Hlbert spaces. 2 Sngle-Class SVMs Sngle-class SVMs were ntrduced [8, 11] t estmate quantles C ß fx 2 jf (x) 2 [ρ; 1[g f an unknwn dstrbutn P n usng kernel expansns. Here, f (x) = ff k(x ;x) ρ; (3) where x 1 ;:::;x m 2are unlabeled data generated..d. accrdng t P. The sngle-class SVM apprxmately cmputes the smallest set C 2 C cntanng a specfed fractn f all tranng examples, where smallness s measured n terms f the nrm n the RKHS H asscated wth k, and C s the famly f sets crrespndng t half-spaces n H. Dependng n the kernel, ths ntn f smallness wll cncde wth the ntutve dea that the quantle estmate shuld nt nly cntan a specfed fractn f the tranng pnts, but t shuld als be suffcently smth s that the same s apprxmately true fr prevusly unseen pnts sampled frm P. Let us brefly descrbe the man deas f the apprach. The tranng pnts are mapped nt H usng the feature map Φ asscated wth k, and then t s attempted t separate them frm the rgn wth a large margn by slvng the fllwng quadratc prgram: fr ν 2 (0; 1], 1 mnmze w2h;ο2rm ;ρ2r 1 2 kwk2 + 1 νm ο ρ (4) subject t hw; Φ(x ) ρ ο ; ο 0: (5) Snce nn-zer slack varables ο are penalzed n the bjectve functn, we can expect that f w and ρ slve ths prblem, then the decsn functn, f (x) =sgn(hw; Φ(x) ρ) wll equal 1 fr mst examples x cntaned n the tranng set, 2 whle the regularzatn term 1 Here and belw, bld face greek character dente vectrs, e.g., ο =(ο 1;:::;ο m) >, and ndces ; j by default run ver 1;:::;m. 2 We use the cnventn that sgn (z) equals 1 fr z 0 and 1 therwse.

4 Fgure 2: Mdels cmputed wth a sngle class SVM usng a Gaussan kernel (2). The three examples dffer n the value chsen fr ff n the kernel - a large value (0.224 tmes the dameter f the hemsphere) n the left fgure and a small value (0.062 tmes the dameter f the hemsphere) n the mddle and rght fgure. In the rght fgure als nn-zer slack varables (utlers) were allwed. Nte that that the utlers (clred blue) n the rght fgure crrespnd t a sharp feature (nn-smthness) n the rgnal surface. kwk wll stll be small. Fr an llustratn, see Fgure Fgure 1. The trade-ff between these tw gals s cntrlled by a parameter ν. One can shw that the slutn takes the frm f (x) =sgn ψ ff k(x ;x) ρ! ; (6) where the ff are cmputed by slvng the dual prblem, 1 mnmze ff ff2rm ff j k(x ;x j ) (7) 2 j subject t 0» ff» 1 νm and ff =1: (8) Nte that accrdng t (8), the tranng examples cntrbute wth nnnegatve weghts ff 0 t the slutn (6). One can shw that asympttcally, a fractn ν f all tranng examples wll have strctly pstve weghts, and the rest wll be zer (the ν-prperty ). In ur applcatn we are nt prmarly nterested n a decsn functn tself but n the bundares f the regns n nput space defned by the decsn functn. That s, we are nterested n f 1 (0), where f s the kernel expansn (3) and the pnts x 1 ;:::;x m 2 are sampled frm sme unknwn hypersurface ρ R d. We want t cnsder f 1 (0) as a mdel fr. In the fllwng we fcus n the case d =3. If we assume that the x are sampled wthut nse frm whch fr example s a reasnable assumptn fr data btaned wth a state f the art 3d laser scannng devce we shuld set the slack varables n (4) and (5) t zer. In the dual prblem ths results n remvng the upper cnstrants n the ff n (8). Nte that sample pnts wth nn-zer slack varable cannt be cntaned n f 1 (0). But als sample pnts whse mage n feature space les abve the ptmal hyperplane are nt cntaned n f 1 (0) (see Fgure 1) we wll address ths n the next sectn. It turns ut that t s useful n practce t allw nn-zer slack varables, because they prevent f 1 (0) frm decmpsng nt many cnnected cmpnents (see Fgure 2 fr an llustratn). In ur experence, ne can ensure that the mages f all sample pnts n feature space le clse t (r n) the ptmal hyperplane can be acheved by chsng ff n the Gaussan kernel (2) such that the Gaussans n the kernel expansn (3) are hghly lcalzed. Hwever, hghly lcalzed Gaussans are nt well suted fr nterplatn the mplct surface decmpses nt several cmpnents. Allwng utlers mtgates the stuatn t a certan extent. Anther way t deal wth the prblem s t further restrct the ptmal regn n feature space. In the fllwng we wll pursue the latter apprach.

5 (ρ+δ*)/ w w Φ( x) x*. ξ */ w (ρ+δ)/ w Φ( ) ξ/ w Fgure 3: Tw parallel hyperplanes hw; Φ(x) = ρ + ff (Λ) enclsng all but tw f the pnts. The utler Φ(x (Λ) ) s asscated wth a slack varable ο (Λ), whch s penalzed n the bjectve functn (9). 3 Slab SVMs A rcher class f slutns, where sme f the weghts can be negatve, s btaned f we change the gemetrc setup. In ths case, we estmate a regn whch s a slab n the RKHS,.e., the area enclsed between tw parallel hyperplanes (see Fgure 3). T ths end, we cnsder the fllwng mdfed prgram: 3 mnmze w2h;ο (Λ) 2Rm ;ρ2r 1 2 kwk2 + 1 νm (ο + ο Λ ) ρ (9) subject t ff ο»hw; Φ(x ) ρ» ff Λ + ο Λ (10) and ο (Λ) 0: (11) Here, ff (Λ) are fxed parameters. Strctly speakng, ne f them s redundant: ne can shw that f we subtract sme ffset frm bth, then we btan the same verall slutn, wth ρ changed by the same ffset. Hence, we can generally set ne f them t zer, say, ff =0. Belw we summarze sme relatnshps f ths cnvex quadratc ptmzatn prblem t knwn SV methds: 1. Fr ff = 0 and ff Λ = 1 (.e., n upper cnstrant), we recver the sngle-class SVM (4) (5). 2. If we drp ρ frm the bjectve functn and set ff = ", ff Λ = " (fr sme fxed " 0), we btan the "-nsenstve supprt vectr regressn algrthm [12], fr a data set where all utput values y 1 ;:::;y m are zer. Nte that n ths case, the slutn s trval, w =0. Ths shws that the ρ n ur bjectve functn cannt be drpped and plays an mprtant rle. P 3. Fr ff = ff Λ =0, the term (ο + ο Λ ) measures the dstance f the pnt Φ(x ) frm the hyperplane hw; Φ(x ) ρ =0(up t a scalng f kwk). If ν tends t zer, ths term wll dmnate the bjectve functn. Hence, n ths case, the slutn wll be a hyperplane that apprxmates the data well n the sense that the pnts le clse t t n the RKHS nrm. Frm the fllwng cnstrants and Lagrange multplers ο ff + hw; Φ(x ) ρ 0; ff 0 (12) ο Λ + ffλ + ρ hw; Φ(x ) 0; ff Λ 0 (13) ο (Λ) 0; f (Λ) 0 (14) 3 Here and belw, the superscrpt (Λ) smultaneusly dentes the varables wth and wthut astersk, e.g., ο (Λ) s a shrthand fr ο and ο Λ.

6 we derve the Lagrangan dual ptmzatn prblem f (9) - (11): 4 mnmze ff2rm subject t and 1 2 j (ff ff Λ )(ff j ff Λ j )k(x ;x j ) ff 0» ff (Λ)» 1 νm ff + ff Λ ff Λ (15) (16) (ff ff Λ )=1; (17) The dual prblem can be slved usng standard quadratc prgrammng packages. The ffset ρ can be cmputed frm the value f the crrespndng varable n the duble dual, r usng the Karush-Kuhn-Tucker (KKT) cndtns, just as n ther supprt vectr methds. Once ths s dne, we can evaluate fr each test pnt x whether t satsfes ff»hw; Φ(x) ρ» ff Λ. In ther wrds, we have an mplct descrptn f the regn n nput space that crrespnds t the regn n between the tw hyperplanes n the RKHS. Fr ff = ff Λ, ths s a sngle hyperplane, crrespndng t a hypersurface n nput space. 5 T cmpute ths surface we use the kernel expansn hw; Φ(x) = (ff ff Λ )k(x ;x): (18) Supprt Vectrs and Outlers In ur dscussn f sngle class SVMs fr surface mdelng we already mentned that we am fr many supprt vectrs (as we want mst tranng pnts t le n the surface) and that utlers mght represent features lke certan sngulartes n the rgnal hypersurface. Here we analyze hw the parameter ν nfluences the SVs and utlers. T ths end, we ntrduce the fllwng shrthands fr the sets f SV and utler ndces: SV := f j hw; Φ(x ) ρ ff» 0g (19) SV Λ := f j hw; Φ(x ) ρ ff Λ 0g (20) OL (Λ) := f j ο (Λ) > 0g (21) It s clear frm the prmal ptmzatn prblem that fr all, ο > 0 mples hw; Φ(x ) ρ ff<0 (and lkewse, ο Λ > 0 mples hw; Φ(x ) ρ ff Λ > 0), hence OL (Λ) ρ SV (Λ). The dfference f the SV and OL sets are thse pnts that le precsely n the bundares f the cnstrants. 6 Belw, jaj dentes the cardnalty f the set A. Prpstn 1 The slutn f (9) (11) satsfes jsv j m jolλ j m ν jolj m jsv Λ j m : (22) The prf s analgus t the ne f the ν-prperty fr standard SVMs, cf. [8]. Due t lack f space, we skp t, and nstead merely add the fllwng bservatns: 4 Nte that due t (17), the dual slutn s nvarant wth respect t the transfrmatn ff (Λ)! ff (Λ) + cnst: such a transfrmatn nly adds a cnstant t the bjectve functn, leavng the slutn unaffected. 5 subject t sutable cndtns n k 6 The present usage dffers slghtly frm the standard defntn f SVs, whch are usually thse that satsfy ff (Λ) > 0. In ur defntn, SVs are thse pnts where the cnstrants are actve. Hwever, the dfference s margnal: () It fllws frm the KKT cndtns that ff (Λ) > 0 mples that the crrespndng cnstrant s actve. () whle t can happen n thery that a cnstrant s actve and nevertheless the crrespndng ff (Λ) s zer, ths almst never ccurs n practce.

7 Fgure 4: Frst rw: Cmputng a mdel f the Stanfrd bunny (35947 pnts) and f a glf club (16864 pnts) wth the slab SVM. The clse up f the ears and nse f the bunny shws the sample pnts clred accrdng t the ceffcents ff ff Λ. Red pnts have negatve ceffcents and green pnts pstve nes. In the rght fgure we shw the bttm f the glf club mdel. The mdel n the left f ths fgure was cmputed wth a dfferent methd [4]. Nte that wth ths methd fne detals lke the fgure three becme vsble. Such detals get leveled ut by the lmted reslutn f the marchng cubes methd. Hwever the nfrmatn abut these detals s preserved and detected n the SVM slutn, as can be seen frm the clr cdng. Secnd rw: In the left and n the mddle fgure we shw the results f the slab SVM methd n the screwdrver mdel (27152 pnts) and the dnsaur mdel (13990 pnts), respectvely. In the rght fgure a clr cdng f the ceffcents fr the rckerarm data set (40177 pnts) s shwn. Nte that we can extract sharp features frm ths data set by flterng the ceffcents accrdng t sme threshld. 1. The abve statements are nt symmetrc wth respect t exchangng the quanttes wth astersks and ther cunterparts wthut astersk. Ths s due t the sgn f ρ n the prmal bjectve functn. If we used +ρ rather than ρ, we wuld btan almst the same dual, the nly dfference beng that the cnstrant (17) wuld have a 1 n the rght hand sde. In ths case, the rle f the quanttes wth and wthut astersks wuld be reversed n Prpstn The ν-prperty f sngle class SVMs s btaned as the specal case where OL Λ = SV Λ = ;. 3. Essentally, f we requre that the dstrbutn has a densty w.r.t. the Lebesgue measure, and that k s analytc and nn-cnstant (cf. [8, 10]), t can be shwn that asympttcally, the tw nequaltes n the prpstn becme equaltes wth prbablty 1. Implementatn On larger prblems, slvng the dual wth standard QP slvers becmes t expensve (scalng wth m 3 ). Fr ths case, we can use decmpstn methds. The adaptatn f knwn decmpstn methds t the present case s straghtfrward, ntcng that the dual f the standard "-SV regressn algrthm [12] becmes almst dentcal t the

8 present dual f we set " =(ff Λ ff)=2 and y = (ff Λ + ff)=2 fr all. The nly dfference s that n ur case, there s a 1 n (17), whereas n the SVR case, we wuld have a 0. As a cnsequence, we have t change the ntalzatn f the ptmzatn algrthm t ensure that we start wth a feasble slutn. As an ptmzer, we used a mdfed versn f lbsvm [3]. Expermental Results In all ur experments we used a Gaussan kernel (2). T render the mplct surfaces,.e., the zer-set f 1 (0), we generated a trangle mesh that apprxmates t. T cmpute the mesh we used an adaptatn f the marchng cubes algrthm [6] whch s a standard technque t transfrm an mplctly gven surfaces nt a mesh. The mst cstly peratns n the marchng cubes algrthm are evaluatns f the kernel expansn (18). T reduce the number f these evaluatns we mplemented a surface fllwng technque that explts the fact that we knw qute sme sample pnts n the surface, namely the supprt vectrs. 7 Sme results can be seen n Fgure 4. Our experments ndcate a nce gemetrc nterpretatn f negatve ceffcents ff ff Λ. It seems that negatve ceffcents crrespnd t cncavtes n the rgnal mdel. The ceffcents seem well suted t extract shape features frm the sample pnt set, e.g., the detectn f sngulartes lke sharp edges r feature lnes whch s an mprtant tpc n cmputer graphcs [7]. We als tred a mult-scale apprach. In ths apprach at frst a rugh mdel s cmputed frm ten percent f the sample pnts usng a slab SVM. Fr the remanng 90% f the sample pnts we cmpute the resdual values,.e., we evaluate the kernel expansn (18) at the sample pnts. Fnally we use supprt vectr regressn (SVR) and the resdual values t derve a new kernel expansn (usng a smaller kernel wdth) whse zer set we use as ur surface mdel. An example hw ths apprach wrks can be seen n Fgure 5. Fgure 5: Frst rw: The mult-scale apprach appled t a knt data set (10000 pnts). The blbby supprt surface (left fgure) was cmputed frm 1000 randmly chsen sample pnts wth the slab SVM. In the mddle fgure we shw a clr cdng f the resdual values f all sample pnts. In the rght fgure we shw the surface that we get after applyng supprt vectr regressn usng the resdual values. 4 Dscussn and Outlk An apprxmate descrptn f the data as the zer set f a functn s nt nly useful as a cmpact representatn f the data. It culd als ptentally be used n tasks such as densng and mage super-reslutn. Gven a nsy pnt x, we can map t nt the RKHS and then prject t nt the hyperplane(s) that we have learnt. We then cmpute an apprxmate pre-mage under Φ t get a nse-free versn f x. 7 In the experments, bth the SVM ptmzatn and the marchng cubes renderng tk up t abut 2 hurs.

9 Sme acqustn devces d nt nly prvde us wth pnts frm a surface embedded n R 3, but als wth the nrmals at these pnts. Usng methds smlar t the nes n [2], t shuld be pssble t ntegrate such addtnal nfrmatn nt ur apprach. We expect that t wll mprve the qualty f the cmputed mdels n the sense that even mre gemetrc detals are preserved. A nce feature f ur apprach s that ts cmplexty depends nly margnally n the dmensn f the nput space (n ur examples ths dmensn was three). Thus the apprach shuld wrk als well fr hypersurfaces n hgher dmensnal nput spaces. Frm an applcatns pnt f vew hypersurfaces mght nt be as nterestng as manflds f hgher c-dmensn. It wuld be nterestng t see f ur apprach can be generalzed t handle als ths stuatn. Acknwledgment We thank Chh-Jen Ln fr help wth lbsvm. The bunny data were taken frm the Stanfrd 3d mdel repstry. The screwdrver, dnsaur and rckerarm data were taken frm the hmepage f Cyberware Inc. References [1] J. Carr, R. Beatsn, J. Cherre, T. Mtchell, W. Frght, B. McCallum and T. Evans. Recnstructn and representatn f 3D bjects wth radal bass functns. Prceedngs f the 28th annual cnference n cmputer graphcs and nteractve technques, 67 76, 2001 [2] O. Chapelle and B. Schölkpf. Incrpratng nvarances n nnlnear SVMs. In T.G. Detterch, S. Becker and Z. Ghahraman, edtrs, Advances n Neural Infrmatn Prcessng Systems 14. MIT Press, Cambrdge, MA, [3] C.-C. Chang and C.-J. Ln. LIBSVM: a lbrary fr supprt vectr machnes, Sftware avalable at cjln/lbsvm. [4] J. Gesen and M. Jhn. Surface Recnstructn Based n a Dynamcal System. Cmputer Graphcs Frum, 21(3): , [5] G. R. G. Lanckret, N. Crstann, P. Bartlett, L. El Ghau, and M. I. Jrdan. Learnng the kernel matrx wth semdefnte prgrammng. JMLR, 5:27 72, [6] T. Lewner, H. Lpes, A. Wlsn and G. Tavares. Effcent Implementatn f Marchng Cubes Cases wth Tplgcal Guarantee Jurnal f Graphcs Tls, 8:1 15, [7] M. Pauly, R. Keser and M. Grss. Mult-scale Feature Extractn n Pnt-Sampled Surfaces. Cmputer Graphcs Frum, 22(3): , [8] B. Schölkpf, J. Platt, J. Shawe-Taylr, A. J. Smla, and R. C. Wllamsn. Estmatng the supprt f a hgh-dmensnal dstrbutn. Neural Cmputatn, 13: , [9] B. Schölkpf, A. J. Smla, and K.-R. Müller. Nnlnear cmpnent analyss as a kernel egenvalue prblem. Neural Cmputatn, 10: , [10] I. Stenwart. Sparseness f supprt vectr machnes sme asympttcally sharp bunds. In S. Thrun, L. Saul, and B. Schölkpf, edtrs, Advances n Neural Infrmatn Prcessng Systems 16. MIT Press, Cambrdge, MA, [11] D. M. J. Tax and R. P. W. Dun. Supprt vectr data descrptn. Machne Learnng, 54:45 66, [12] V. N. Vapnk. The Nature f Statstcal Learnng Thery. Sprnger Verlag, New Yrk, 1995.