Sparse Kernel Ridge Regression Using Backward Deletion

Transcription

1 Sparse Kerne Rdge Regresson Usng Bacward Deeon Lng Wang, Lefeng Bo, and Lcheng Jao Insue of Inegen Informaon Processng 7007, Xdan Unversy, X an, Chna {wp, bf08}@63.com Absrac. Based on he feaure map prncpe, Sparse Kerne Rdge Regresson (SKRR) mode s proposed. SKRR obans he sparseness by bacward deeon feaure seecon procedure ha recursvey removes he feaure wh he smaes eave-one-ou score un he sop creron s sasfed. Besdes good generazaon performance, he mos compeng propery of SKRR s raher sparse, and moreover, he erne funcon needs no o be posve defne. Expermens on synhec and benchmar daa ses vadae he feasby and vady of SKRR. Inroducon Regresson probem s one of he fundamena probems n he fed of supervsed earnng. I can be hough of as esmang he rea vaued funcon from a sampes se of nose observaon. A very successfu approach for regresson s Suppor Vecor Regresson (SVR) [-] ha aemps o smuaneousy mnmze emprca rs and confdence nerva, eadng o good generazaon. Due o ε -nsensve oss funcon, SVR obans a sparse mode (predcon for a new npu ony needs he subse of ranng sampes). hough very successfu, SVR aso has some dsadvanages [3]: he souon s usuay no very sparse and he predcon speed for a new npu s sgnfcany sower han ha of some oher earnng machnes such as Neura Newors [4-5]. Kerne funcon mus sasfy Mercer s condon. I s we nown ha dfferen erne funcons w nduce dfferen agorhms, achevng dfferen performance. However, erne funcons n SVR mus sasfy Mercer s posve defne condon, whch ms he usabe ernes. In order o dea wh he probems menoned above, Reevance Vecor Machnes (RVM) [3] s proposed, whch s very eegan and obans a hgh sparse souon. However, RVM needs o sove near equaons, whose cos s very expensve, and herefore, s no appcabe o arge scae probems. In hs paper, we propose a new earnng mode, Sparse Kerne Rdge Regresson (SKRR) o overcome he above probems. In SKRR, sampes are mapped no he feaure space whose dmenson s equa o he sampe sze, and hen Rdge Regresson (RR) [6] s mpemened n he feaure space. When he ranng process s compeed, a bacward deeon feaure seecon procedure s apped o oban a sparse mode. Q. Yang and G. Webb (Eds.): PRICAI 006, LNAI 4099, pp , 006. Sprnger-Verag Bern Hedeberg 006

2 366 L. Wang, L. Bo, and L. Jao Besdes good generazaon performance, he mos compeng propery of SKRR s s sparseness ha s comparabe wh ha of RVM. Anoher advanage of SKRR s ha he erne funcon needs no o be posve defne. Expermens on synhec and benchmar daa ses assess he feasby and vady of SKRR. Kerne Rdge Regresson Le {( x, y ), L,( x, y )} be emprca sampes se drawn from y = f( x ) + ε, =,, L,, () where y s corruped by addve nose ε, whose dsrbuons are usuay unnown. Learnng ams o nfer he funcon f ( x ) from he fne daa se {( x, y ), L,( x, y )}. A cassca mehod for he probem s Rdge Regresson (RR), whch s an exenson of near regresson by addng a quadrac penazng erm: ( ) wˆ = arg mn w x + w+ y + λ w, () = where λ s a fxed posve consan, caed reguarzaon coeffcen. Equaon () can be rewren as ( ( λ ) ) w ˆ = arg mn w X X+ I w w X y, (3) ' x, w where X = M, w = M. ', x w + Rdge regresson [6] s a we-nown approach for he souon of regresson probems, whch has a good generazaon performance as we as SVR, and he mode does no need he erne funcon sasfyng Mercer s condon, moreover, here s an effcen eave-one-ou cross-vadaon mode seecon mehod. In order o mae RR appcabe o he nonnear probems, we generaze by a feaure map. Defne a ( xx, ) =,, L,, as shown vecor made up of a se of rea-vaued funcons { } [ ] z = ( x, x ), L, ( x, x ), (4) ' where ( xx, ) s erne funcon ha needs no o be posve defne. We ca { [ (, ),, (, ) ], n z z xx xx x } F = = L R (5) { (, ),,(, ) (, ),, (, ) } feaure space. In parcuar, z y z y z = [ x x x x ] L L are emprca sampes n he feaure space. Subsung z for x, we have he Kerne Rdge Regresson (KRR) [7] n he feaure space F

3 Sparse Kerne Rdge Regresson Usng Bacward Deeon 367 ( ( λ ) ) w ˆ = arg mn w Z Z+ I w w Z y, (6) ' z, w where Z = M, w = M. he correspondng decson funcon s ', z w + f( x) = w( x, x ) + w +. (7) = From equaon (6) and (7), we can oban w=[z ˆ Z+ λi] Z Y. (8) 3 Sparse Kerne Rdge Regresson For a suabe λ, KRR has a good generazaon performance. Is ey dsadvanage s no sparseness, whch seems o prohb s appcaon n some feds. In hs paper, we consder how o deee redundan feaures and smuaneousy eep good generazaon performance a an accepabe compuaona cos. 3. Feaure Seecon by Bacward Deeon In order o oban he sparseness, a bacward deeon procedure s mpemened afer he ranng process. Bacward deeon procedure recursvey removes he feaure wh he smaes eave-one-ou score un he sop creron s sasfed. Le H = ( Z Z+λI ) and b = Z y, hen equaon (6) can be rewren as ( L ( )) w ˆ = arg mn = w Hw w b. (9) h When he feaure s deeed a he h eraon, e H represen he sub-marx h formed by omng he row and coumn of H. Le R represen he nverse of H, (8), we have w he weghs and f he opma vaue of L. Accordng o equaon ( ) f = b R b. (0), P {} h where P s a se of remanng feaures (varabes) a he eraon. In erms of a ran- updae [8-9], R and w can be formuaed n equaon () and () (see Appendx for deas), where R represens he nverse of H. =,, P {}. ()

4 368 L. Wang, L. Bo, and L. Jao n ( w ) = ( ), { } R b P. P { } ( ) R () ogeher wh w = R b, equaon () s smpfed as P P ( w ) = w w, P { }. (3) Subsung equaon () no (0), we oban ( ) R f = b b + b b = + P f, P, P P By he vrue of b b = w, equaon (4) s ransaed no ( w ). (4) Δ f = f f =. (5) We ca Δ f eave-one-ou score. A each eraon, we remove he feaure wh he smaes eave-one-ou score. he feaure o be deeed can be obaned by he foowng expresson. s = arg mn Δ f. (6) P { + } ( ) (, s) Z (,) s w (, s) H Fg.. Dsrbuon of parameers afer he h s feaure was deeed Fgure shows he dsrbuon of parameers afer he s h feaure was deeed. Noe ha he ( + ) h varabe s bas ha s reserved durng he feaure seecon process.

5 Sparse Kerne Rdge Regresson Usng Bacward Deeon 369 A he h eraon, he oa ncrease of oss funcon L s where Δ f = bw L. (7) op P op L s he mnmum of equaon (9). We ermnae he agorhm f op Δf ε L. (8) where ε s a sma posve number. Accordng o he dervaon above, Bacward Deeon Feaure Seecon (BDFS) can be descrbed as he foowng Agorhm : Agorhm. BDFS. Se P = {,, L, }, R = H, w = R b;. For = o, do: ( w ) 3. (a) s = arg mn ; P { + } ( ) (,) s R s s R = R,, P s 4. (b) ( ) ( ) {} (,) s s 5. (c) ( w ) = w ws, ss P { s} ; 6. (, s) (d) P = P { s}, (, s) R = R, w + = w. 7. (e) IF ( bw + P bw) εbw, Sop. 8. End For 9. End Agorhm 3. Mode Seecon here exs free parameers ncudng erne parameer and reguarzaon parameer n SKRR. In order o oban good generazaon, s needed o choose he suabe parameers. Cross-vadaon s a mode seecon mehod ofen used n esmang he generazaon performance of sasca cassfers. 0-fod cross-vadaon s ofen used n some erne-based earnng agorhms such as SVMs. Leave-one-ou cross vadaon s he mos exreme form of cross-vadaon. Leave-one-ou cross vadaon error s an aracve mode seecon creron snce provdes an amos unbased esmaor of generazaon performance []. However, hs mehod s rarey adoped n he erne machnes because s compuaonay expensve. Forunaey for SKRR, here s an effcen mpemenaon for eave-one-ou cross 3 vadaon ha ony ncurs a compuaona cos of O ( ) [0]. Le E( Γ ) be eave-oneou cross vadaon error, where Γ denoes free parameers. ss ;

6 370 L. Wang, L. Bo, and L. Jao Lemma. [-]. E( Γ ) = B( Γ)( I-A) y, where ( Γ) he h enry α ( ) ( Γ ) = ( +λ ) A Z Z Z I Z. ( ) Γ, ( ) Le he snguar vaue decomposon of Z be B s a dagona marx wh α Γ beng he h enry of Z = UDV, (9) where UV, are orhogona marces, D s a dagona marx. Subsung equaon (9) no A ( Γ), can be smpfed as ( Γ ) = ( +λ ) A UD D D I D U, (0) where DD+ λi s a dagona marx. Hence for dfferen λ, we ony need o perform once marx decomposon. For cary, here we gve he dea seps of eave-one-ou mode seecon agorhm n whch σ represens erne parameer: Agorhm. LOO Mode Seecon For σ, =,, L, q Z = UDV ; For λ, =,, L, p ( α = UD DD+ λ I) DU ; ( ) B = α ; r( ) = B ( y A y ) { } = r( ) = E End for End for op op ( λ, σ ) = argmn( E )., hus, accordng o he anayss n secon 3. and 3., SKRR can be descrbed n he foowng Agorhm 3: Agorhm 3. SKRR. Le λ be nose varance, and choose he erne parameers usng LOO mode seecon agorhm;. ran KRR usng he seeced parameers; 3. Impemen BDFS for KRR; 4. Re-esmae λ usng LOO mode seecon agorhm; 5. Re-ran KRR on he smpfed feaures.

7 Sparse Kerne Rdge Regresson Usng Bacward Deeon 37 4 Smuaon In order o evauae he performance of he proposed agorhm, we performed SKRR on hree daa ses: Snc, Boson Housng and Abaone daa ses, and compared s performance wh ha of KRR, SVR, and RVM. For he sae of comparson, dfferen agorhms use he same npu sequence. he eemens of Gram Marx are consruced usng Gaussan erne of he form ( xy, ) = exp. For SVR and x y σ RVM, we uze 0-fod cross vadaon procedure o choose he free parameers. For a daases, parameer ε n BDFS s se 0.0. Large amouns of expermens show ha s a good seecon. 4. oy Expermen he Snc funcon y = sn( x) / x+ σ N(0,) s a popuar choce o usrae suppor vecor machnes regresson. We generae ranng sampes from Snc funcon a 00 equay-spaced x -vaue n [ 0,0] wh added Gaussan nose of sandard devaon 0.. Resus are averaged over 00 random nsanaons of he nose, wh he error beng measured over 000 nose-free es sampes n [ 0,0]. he decson funcons and suppor vecors obaned by SKRR and SVR wh ε = 0. are shown n Fgure. Suppor vecors number and mean square error are summarzed n abe. For Snc daa se, SKRR ouperforms oher hree agorhms. SKRR and RVM oban smar suppor vecors number ha s sgnfcany ess han ha of SVR. abe. Generazaon error obaned by he four agorhms on Snc daa se. MSE denoes mean square error and NSV denoes suppor vecors number SKRR KRR SVR RVM MSE 0.85 ± ± ± ± 0.47 NSV 7.0 ± ± ± ± 0.38 Fg.. Suppor vecors and decson boundary obaned by SKRR and SVR. Rea red nes denoe he decson boundary and do nes denoe he Snc funcon; crces denoe he suppor vecors.

8 37 L. Wang, L. Bo, and L. Jao 4. Benchmar Comparson Boson Housng and Abaone daa ses ha come from SALOG COLLECION [3] are popuar choces o evauae he performance of agorhms. Boson Housng daa se ncudes 506 exampes, 3 arbues of each exampe and Abaone daa se ncudes 477 exampes, 7 arbues of each exampe. For he Boson Housng daa se, we average our resus over 00 random sps of he fu daase no 48 ranng sampes and 5 esng sampes. For he Abaone daa se, we average our resus over 0 random sps of he moher daase no 000 ranng sampes and 377 esng sampes. Before expermens, we scae a ranng sampes no [-, ] and hen adus esng sampes usng he same near ransformaon. he resus are summarzed n abe and 3. abe. Generazaon error obaned by he four agorhms on Boson Housng daa se. MSE denoes mean square error and NSV denoes suppor vecors number. SKRR KRR SVR RVM MSE 0.05 ± ± ± ± 6.8 NSV 5.3 ± ± ± ±.49 From he expermena resus (abe, and 3), we observe he foowng SKRR, KRR and RVM oban smar performance and are sghy superor o SVR. Boh SKRR and RVM are raher sparse. Suppor (reevance) vecors number of SKRR and RVM s much ess han ha of SVR. abe 3. Generazaon error obaned by he four agorhms on Abaone daa se. MSE denoes mean square error and NSV denoes suppor vecors number. SKRR KRR SVR RVM MSE 4.64 ± ± ± ± 0. NSV 8.90 ± ± ± ±.45 5 Concuson Bacward deeon feaure seecon provdes a sae-of-he-ar oo ha can deee redundan feaures and smuaneousy eep good generazaon performance a an accepabe compuaona cos. Based on BDFS, we propose SKRR ha obans hgh sparseness. Furher appcaon of BDFS ncudes condensng he wo-sage RBF Newors. Appcaon o near regresson s n progress. Acnowedgmens hs wor was suppored by he Naona Naura Scence Foundaon of Chna under gran No and he Naona Grand Fundamena Research 973 Program of Chna under Gran No. 00CB

9 Sparse Kerne Rdge Regresson Usng Bacward Deeon 373 References. Vapn, V: he Naure of Sasca Learnng heory. New Yor Sprnger-Verag (995). Vapn, V: Sasca Learnng heory. New Yor Wey-Inerscence Pubcaon (998) 3. ppng, M.E: he Reevance Vecor Machne. Proc. Advances n Neura Informaon Processng Sysems, Soa,S., Leen,. and Müer K.-R. eds. Cambrdge, Mass MI Press (00) Nea, R: Bayesan Learnng for Neura Newors. New Yor Sprnger Verag (996) 5. Rpey R: Paern Recognon and Neura Newors. Cambrdge, U.K. Cambrdge Unv. Press (996) 6. Hoer, A. and Kennard, R: Rdge Regresson: Based Esmaon for Nonorhogona Probems. echnomercs (970) Saunder C. and Gammerman A: Rdge regresson earnng agorhm n dua varabes. In: Shav J. edor. Machne earnng Proceedngs of he 5 h Inernaona Conference, Morgan Kaufmann (998) 8. Soer, J. and Bursch, R: Inroducon o Numerca Anayss. Sprnger Verag, New Yor, Second Edon (993) 9. Bo, L.F., Wang, L., and Jao, L.C.: Sparse Gaussan processes usng bacward emnaon. Lecure Noes n Compuer Scence 397 (006) Bo, L.F., Wang, L., and Jao, L.C.: Feaure scang for erne Fsher dscrmnan anayss usng eave-one-ou cross vadaon. Neura Compuaon 8(4) (006) Aen, D.M: he reaonshp beween varabe seecon and predcon. echnomercs 6 (974) 5-7. Gavn C.C.: Effcen eave-one-ou cross-vadaon of erne fsher dscrmnan cassfers. Paern Recognon 36() (003) Mche, D, Spegehaer, D. J., and ayor, C.C.: Machne Learnng, Neura and Sasca Cassfcaon. Prence Ha, Engewood Cffs, N.J., 994, Daa avaabe a anonymous fp: fp.ncc.up.p/pub/saog/. Appendx he marx nverson formua for a symmerc marx wh boc sub-marces s gven n he foowng Lemma. Lemma [8]: Gven nverbe marces A and C, and marx B, he foowng equay hods: A B A + A BE B A E BC = B C C B E C + C B D BC, (A.) D= A BC B where E= C B A B. Equaon () can be derved n erms of Lemma. We assume here ha B C s h he feaure o be deeed. hus n our probem, R corresponds o A, and

10 374 L. Wang, L. Bo, and L. Jao corresponds o E, and ( ) R corresponds o. Observng he op ef boc of he as par n equaon (A.), we have whch corresponds o Hence, we have A + A BE E B A E +,, P {} =,, P {} A BE, P { }, (A.). (A.3). (A.4)