MaxMinOver Regression: A Simple Incremental Approach for Support Vector Function Approximation

Transcription

1 MaxMnOver Regresson: A Smple Incremental Approach for Support Vector Functon Approxmaton Danel Schneegaß,2,KaLabusch, and Thomas Martnetz Insttute for Neuro- and Bonformatcs Unversty at Lübeck, D Lübeck, Germany martnetz@nformatk.un-luebeck.de 2 Informaton & Communcatons, Learnng Systems Semens AG, Corporate Technology, D-8739 Munch, Germany danel.schneegass.ext@semens.com Abstract. The well-known MnOver algorthm s a smple modfcaton of the perceptron algorthm and provdes the maxmum margn classfer wthout a bas n lnearly separable two class classfcaton problems. In [] and [2] we presented DoubleMnOver and MaxMnOver as extensons of MnOver whch provde the maxmal margn soluton n the prmal and the Support Vector soluton n the dual formulaton by dememorsng non Support Vectors. These two approaches were augmented to soft margns based on the ν-svm and the C2-SVM. We extended the last approach to SoftDoubleMaxMnOver [3] and fnally ths method leads to a Support Vector regresson algorthm whch s as effcent and ts mplementaton as smple as the C2-SoftDoubleMaxMnOver classfcaton algorthm. Introducton The Support-Vector-Machne SVM [4], [5] s a very effcent, unversal and powerful tool for classfcaton and regresson tasks e.g. [6], [7], [8]. A major drawback, partcularly for ndustral applcatons where easy and robust mplementaton s an ssue, s ts complcated tranng procedure. A large Quadratc-Programmng problem has to be solved, whch requres sophstcated numercal optmsaton routnes whch many users do not want or cannot mplement by themselves. They have to rely on exstng software packages, whch are hardly comprehensve and, n some cases at least, error-free. Ths s n contrast to most Neural Network approaches where learnng has to be smple and ncremental almost by defnton. The teratve and ncremental nature of learnng n Neural Networks usually leads to smple tranng procedures whch can easly be mplemented. It s desrable to have smlar tranng procedures also for the SVM. Several approaches for obtanng more or less smple ncremental tranng procedures for Support Vector classfcaton and regresson have been ntroduced so far [9], [0], [], [2]. We want to menton n partcular the Kernel-Adatron by Fress, Crstann, and Campbell [9] and the Sequental-Mnmal-Optmsaton algorthm SMO by Platt [0]. Ths s the most wdespread teratve tranng procedure for SVMs. It s fast and robust, but t s stll not yet of a pattern-by-pattern nature and well suted as a S. Kollas et al. Eds.: ICANN 2006, Part I, LNCS 43, pp , c Sprnger-Verlag Berln Hedelberg 2006

2 MaxMnOver Regresson 5 startng pont for an onlne learnng scheme. It s not yet as easy to mplement as one s used from Neural Network approaches. Accordng to ths goal n [2] and [] we had revsted and extended the MnOver algorthm. The so-called DoubleMaxMnOver [3] method whch we brefly revst n ths paper as the combnaton of DoubleMnOver and MaxMnOver converges provable to the Support Vector soluton n the dual representaton for classfcaton tasks. Moreover, the angle γ t between the correct soluton w and the nterm soluton w t after t steps s bounded by Ot. Furthermore we ntroduced a soft margn approach based on the C2-SVM error model. In a straghtforward way ths model can be adapted for regresson [3]. We consequently show that we smlarly obtan our regresson algorthm as an extenson of DoubleMaxMnOver for classfcaton as well. We gve a proof for ts convergence propertes and consder benchmark results. 2 The Support Vector Machne The SVM represents a lnear classfer or functon approxmator, respectvely. For gven sets of nput vectors X = {x,...,x l } and output vectors Y = {y,...,y l } the SV soluton for the nference of fx =y s always the best one n the sense that n classfcaton the margn between the separatng hyperplane and the two classes s largest and n regresson the obtaned curve s the flattest one. In both cases one only needs a few nput vectors, the so-called Support Vectors, to descrbe the soluton. Ths property s very mportant. The lower the number of Support Vectors, the better the expected generalsaton capablty [4]. The goal s to solve the optmsaton problem 2 wt w =mn {,...,l} : y w T x b n classfcaton, whch s apparently equvalent to the maxmsaton of the margn holdng w,and 2 wt w =mn {,...,l} : w T x b y ɛ. n regresson tasks. Note that classfcaton and regresson are geometrcally analogous n the followng sense. Whle n classfcaton the Support Vectors are the ones lyng on the margn, that s havng smallest dstance to the separatng hyperplane, n regresson the Support Vectors le on the edges of the ɛ-tube around the functon values. In classfcaton all other vectors are located more or less far away from the margn. In regresson they le wthn the ɛ-tube. Hence SV classfcaton can be seen as the perspectve from nsde to outsde the hyperplane area whle SV regresson s the other way around.

3 52 D. Schneegaß, K. Labusch, and T. Martnetz An mportant advantage of SVMs s furthermore that the classfer or functon approxmator fx =w T x b can be formulated n terms of the observatons,.e. as l fx = = α x T x b. Therefore, t s possble to replace the nner product x T z by a postve defnte Kernel Kx, z = Φx, Φz whch leads to the mplct use of arbtrary feature spaces, whose defnton has not necessarly to be known, makng profoundly non-lnear problems lnear. 3 The C2-SoftDoubleMaxMnOver Algorthm for Classfcaton The DoubleMnOver algorthm [] as the frst extenson of MnOver s a smple maxmal margn learnng machne. Startng from any nterm soluton w t the method selects two vectors each of both classes havng the smallest margn to the hyperplane defned by V = {v w T v b =0}, thats a,mn =argmn,y=a afx, respectvely. Afterwards the Lagrange-coeffcents α wll be ncreased by to change the drecton of the weght vector gven by x : fx =w T x = l = y α Kx, x towards these two nput vectors and ncrease ther margns. By nducton t can be seen that w t ndeed tends to w whle t. It has been shown [5] that the angle γ t between the correct soluton w and the nterm soluton w t after t steps s bounded by Ot. Furthermore the tme complexty per teraton s Ol, wherel s the number of nput vectors. In order to acheve the dual SV soluton, not to decrease the convergence speed and to hold the optmsaton constrants, n DoubleMaxMnOver the vectors wth the largest margn wll be decreased, f t s known that they cannot be Support Vectors whle the ones wth smallest margn wll be ncreased twce. In addton we ntroduced a soft margn approach workng wth an extended kernel K x, x j =Kx, x j + δ,j C. It can be seen that ndeed the data remans lnearly separable by constructon [3] wthn ths extended feature space. Later we wll see that all these statements can straghtforwardly be adopted for our regresson approach. The combnaton of ths soft margn, the DoubleMnOver, and the MaxMnOver approaches fnally leads to the C2-SoftDoubleMaxMnOver [,2,3] method, whch s as fast as the LbSVM toolbox on standard benchmarks and performs comparable results. 4 The MaxMnOver Regresson Approach More general than the classfcaton task s the one of regresson. Now the goal s to approxmate a functon gx by usng fx = l α Kx, x b. = Ths can be nterpreted as lnear regresson wthn an approprate feature space, agan defned by the Kernel K. To get a scope for the mnmsaton of w the user defnes

4 MaxMnOver Regresson 53 an ɛ-tube around the real functon values y = gx. As n our soft margn classfcaton approach we consequently use the C2-SVM model for regresson. If C the Support Vector soluton s n a way the smplest or the flattest one whle makng no more error than ɛ. The Support Vectors are the vectors wth Lagrange-coeffcents α 0 lyng on the edges of the ɛ-tube. The ntroducton of a fnte C s not only necessary as regularsaton parameter, but also f t cannot be guaranteed that a regresson wth an errorofatmostɛ s achevable at all. Fortunately the geometrcal nterpretaton s smlar to the one n the C2-SVM classfcaton, so t s not dffcult to see that also n the regresson case an unequvocal Support Vector soluton exsts, whch can be nterpreted Algorthm. The MaxMnOver Regresson Algorthm Requre: gven set of normed nput vectors X = {x,y, {,...,l}} wth normed functon values and parameters C and ɛ Ensure: calculates the mnmal hyperplane regressng the nput data gven n dual representaton set : α 0, t 0, f x P l j= αjkxj, x+ α C q R max,j,y =,y j = Kx, x 2Kx, x j+kx j, x j whle the desred precson s not reached do set t t + fnd mn = arg mn fx y fnd max = arg max fx y fnd mnnsv = arg max,α >0 fx y fnd maxnsv = arg mn,α <0 fx y f fx max y max fx mn y mn > 2ɛ then f fx mnn SV y mnn SV fx mn y mn > 4R ɛ 2 +4ɛ then t set α mn α mn + +mn t t,α mnn SV set α mnn SV α mnn SV mn t,α mnn SV else set α mn α mn + t end f f fx max y max fx maxn SV y maxn SV > 4R ɛ 2 +4ɛ then set α max α max t mn t, α maxn SV set α maxn SV α maxn SV +mn t, α maxn SV else set α max α max t end f else f fx mn y mn fx mnn SV y mnn SV > 4R ɛ 2 +4ɛ set α mnn SV α mnn SV mn t,α mnn SV end f f fx max y max fx maxn SV y maxn SV > 4R ɛ 2 +4ɛ set α maxn SV α maxn SV +mn t, α maxn SV end f end f end whle set b 2 fx mn y mn +fxmax y max t t t then then

5 54 D. Schneegaß, K. Labusch, and T. Martnetz as the soluton wth at most ɛ error wthn the extended feature space defned by K x, z =Kx, z + δx,z C. Ths s as reasonable as the statement that the precondtoned lnear equaton system K + C E l α = y, whch s used n kernalzed Rdge Regresson [3], has a soluton for all but fntely many C<, even f the bas b =0 and ɛ =0. As a frst approach consder the drect adaptaton of the DoubleMnOver algorthm. As n the classfcaton problem, where we choose one nput vector each of both classes wth the smallest margn, we now choose the two nput vectors makng the largest postve and negatve regresson error, respectvely. Apparently, usng an approprate step wdth, the algorthm converges to any feasble soluton, that s all nput vectors le wthn the ɛ-tube and the above constrant holds true by constructon n each teraton. Stll f the Lagrange-coeffcent of any of the nput vectors wll be ncluded faulty as potental Support Vectors, we need an nstrumentaton to turn back ths decson, f they are ndeed non Support Vectors. Once agan as n the classfcaton task, where we choose the nput vectors each of both classes wth the largest margn, we now choose the two vectors lyng furthest away from ts edge of the ɛ-tube and dememorse them by controllng the Lagrange-coeffcents back wthout loosng convergence speed see algorthm. Consequently ths s the MaxMnOver Regresson algorthm. The dffcult lookng dememorsaton crteron wll be explaned n the next secton. 4. On the Convergence of MaxMnOver Regresson Fg.. Constructon of the classfcaton problem. The black dots are the true samples whch we want to approxmate. By shftng them orthogonal to the SVM ft curve n both senses of drecton and assgnng dfferent classes brght x and dark gray x dots we construct a classfcaton problem whose maxmal margn separatng hyperplane s exactly the regresson ft. In the followng wthout loss of generalty we assume C and suppose that a soluton wth no more error than ɛ exsts. Otherwse we choose an approprate C>0and reset K x, z =Kx, z+ δx,z C. Frst of all, the Support Vector soluton and only the

6 MaxMnOver Regresson 55 SV soluton s a fxed pont of the algorthm. Ths can be seen as follows. Once we have reached the SV soluton, the outer condton fx max y max fx mn y mn > ɛ + ɛ of the algorthm s evaluated to false, all ponts have to be wthn the ɛ-tube. The left hand sde of the nner condtons s always 0, because only Support Vectors have non-zero Lagrange coeffcents and all Support Vectors le exactly on the edge of the ɛ-tube, whle the rght hand sde s always postve. Hence nothng would change and the algorthm has fnshed. On the other hand, t can further be seen that no other confguraton of α can be a fxed pont. If any par of vectors le outsde of the ɛ-tube, then the outer condton would be fulflled. Otherwse there has to be at least one vector wthn the ɛ-tube and not on ts edge havng non-zero Lagrange coeffcent and therefore leadng to a postve fulfllment of the nner condton after a fnte number of teratons. Now we show that the MaxMnOver Regresson algorthm ndeed approaches the Support Vector soluton, by ascrbng the regresson problem to a classfcaton problem. We construct the classfcaton problem wthn the extended dmz+-dmensonal space, where Z s the feature space and the addtonal dmenson represents the functon values y, and demonstrate that the classfcaton and the regresson algorthms both work unformly n the lmt. Let X = {x,...,x l } be the set of nput vectors wth ts functon values Y = {y,...,y n } and w the Support Vector soluton of the regresson problem. Of course, f one uses a non-lnear feature space, then x has to be replaced by Φx or Φ x, respectvely. We construct the set ˆX = {x, x,...,x l, x l } from X by substtutng x = x + +ɛ w y w 2 + x x = +ɛ w y w 2 + and assgnng the classes y a = a. The Support Vector soluton of ths classfcaton w problem s ŵ = apart from ts norm wth functonal margn, because mn,y max yŵ T x y b b =mn,y max yw T x y b b + + ɛ 2 = ɛ ++ɛ = and for each vector ˆv ŵ wth ˆv = ŵ t holds both ˆv T ŵ +ɛ w 2 + < ŵ T ŵ +ɛ w 2 + =+ɛ compare eqn. part 2 and at least for the Support Vectors ether v T x y b ɛ or y v T x + b ɛ usng any bas b compare eqn. part, because w s the Support Vector soluton of the actual regresson problem.

7 56 D. Schneegaß, K. Labusch, and T. Martnetz Now suppose an nterm soluton w. Then MaxMnOver for classfcaton and for regresson choose the same vectors n the next teraton. It holds for the frst two fndstatements of algorthm arg mn w T x y =argmn w T x y + S 3 =argmnŵ T x arg max w T x y = arg max =argmax =argmn w T x y S 4 ŵ T x ŵ T x wth S = + ɛwt w + w 2 + and further for the the second two statements concernng the dememorsaton of non Support Vectors arg max,α wt x y = arg max w T x y + S 5 >0,α >0 =arg max,α ŵt x >0 arg mn,α wt x y =arg mn <0,α <0 =arg mn,α ŵt x <0 =arg max ŵ T x,α. <0 w T x y S 6 Note that the constrants α 0, respectvely α 0 mplctly hold for eq. 3, respectvely eq. 4 whle for eqn. 5 and 6 t s necessary to choose only potental Support Vectors of the correct classes for dememorsaton. The presented nterrelatonshp mples that, f the classfcaton algorthm wll ncrement or decrement some y α, then the regresson algorthm does so as well. But whle ŵ classfcaton ncreases lnearly n tme and converges to ŵ apart from ts norm, ŵ regresson must not ncrease arbtrarly. There exsts a tme t start <t, from whch on C,C 2 R : C t< ŵ classfcaton l+ <C 2 t wth C <C 2.Butfor ŵ regresson s ŵ regresson l+ = mplctly gven. Hence ŵ regresson has to be constraned and therefore we choose an approprate step wdth of order O t nstead of normalsng after each teraton. Although the harmonc seres dverges and hence any possble soluton vector wthn the feature space s reachable, we want to emphasze that the convergence speed can be tuned sgnfcantly by normalsng Y or choosng an approprate constant factor for the step wdth.

8 MaxMnOver Regresson 57 We derve the crteron for dememorsaton of non Support Vectors from the classfcaton method, where R 2 classfcaton max,j,y =,y j= x x j 2 has already been proven [,2]. As far ths crteron must only be an upper bound, we estmate Rregresson 2 =max x,j xj + +ɛ y w 2 + y j +ɛ w 2 + w w 2 Rclassfcaton ɛ2 +4ɛ + w 2 + R 2 classfcaton ɛ 2 +4ɛ. Note that we do not need to know the soluton w. Hence t s possble to apply ths estmaton n practce. 5 Expermental Results on Artfcal Data To evaluate the MaxMnOver Regresson algorthm we constructed randomly generated artfcal datasets and modfed the varance, the regresson error, the parameters C and ɛ, the number of teraton steps, the dstrbuton of the data ponts, the number of tranng examples, and the dmenson of the feature space. The table shows the averaged results of these evaluatons. It can be seen that the regresson error s comparable to the one acheved by the LbSVM Toolbox. The hgher the number of teratons the better s the performance of our method and the closer to the results of the LbSVM, whose step wdth was not changed. Only the number of Support Vectors s sometmes dfferent. More teraton steps n both methods should lead to a convergence of the number of Support Vectors to each other. Table. Averaged regresson results obtaned wth MaxMnOver Regresson B on artfcal data. For comparson averaged results obtaned wth the ɛ-svr of the LbSVM Toolbox A are lsted. The smple MaxMnOver Regresson algorthm acheves comparable results wth a few tranng steps. Capton: w norm of weght vector, L 2 squared error, L 2,ɛ squared error toleratng error ɛ, SV number of Support Vectors Parameters Results steps dstr number w A w B L 2,A L 2,B L 2,ɛ,A L 2,ɛ,B SV A SV B w A w B w A unform normal

9 58 D. Schneegaß, K. Labusch, and T. Martnetz 6 Conclusons Regresson s an mportant mathematcal problem whch occurs n a wde varety of practcal applcatons. Support Vector regresson acheves results wth a hgh generalsaton capablty. Dfferent complcated Support Vector regresson approaches have been ntroduced n the past. The man goal of ths paper s hence to show that even Support Vector regresson can be dealt wth n a smple way wth the MaxMnOver Regresson approach. In benchmarks the method acheves performances as good as the LbSVM Toolbox. Both, ts smplcty and ts good performance makes ths approach nterestng for mplementatons n ndustral envronments. References. Martnetz, T., Labusch, K., Schneegass, D.: Softdoublemnover: A smple procedure for maxmum margn classfcaton. Proc. of the Internatonal Conference on Artfcal Neural Networks Martnetz, T.: Maxmnover: A smple ncremental learnng procedure for support vector classfcaton. Proc. of the Internatonal Jont Conference on Neural Networks IEEE Press Schneegass, D., Martnetz, T., Clausohm, M.: Onlnedoublemaxmnover: A smple approxmate tme and nformaton effcent onlne support vector classfcaton method. Proc. of the European Symposum on Artfcal Neural Networks 2006 n preparaton 4. Cortes, C., Vapnk, V.: Support-vector networks. Machne Learnng Vapnk, V.: The Nature of Statstcal Learnng Theory. Sprnger-Verlag, New York LeCun, Y., Jackel, L., Bottou, L., Brunot, A., Cortes, C., Denker, J., Drucker, H., Guyon, I., Muller, U., Sacknger, E., Smard, P., Vapnk, V.: Comparson of learnng algorthms for handwrtten dgt recognton. Int.Conf.on Artfcal Neural Networks Osuna, E., Freund, R., Gros, F.: Tranng support vector machnes:an applcaton to face detecton. CVPR Schölkopf, B.: Support vector learnng Fress, T., Crstann, N., Campbell, C.: The kernel adatron algorthm: a fast and smple learnng procedure for support vector machne. Proc. 5th Internatonal Conference on Machne Learnng Platt, J.: Fast Tranng of Support Vector Machnes usng Sequental Mnmal Optmzaton. In: Advances n Kernel Methods - Support Vector Learnng. MIT Press Keerth, S.S., Shevade, S.K., Bhattacharyya, C., Murthy, K.R.K.: A fast teratve nearest pont algorthm for support vector machne classfer desgn. IEEE-NN L, Y., Long, P.: The relaxed onlne maxmum margn algorthm. Machne Learnng Crstann, N., Shawe-Taylor, J.: Support Vector Machnes And Other Kernel-based Learnng Methods. Cambrdge Unversty Press, Cambrdge Vapnk, V.N.: Statstcal Learnng Theory. John Wley & Sons, Inc., New York Martnetz, T.: Mnover revsted for ncremental support-vector-classfcaton. Lecture Notes n Computer Scence