Sparse Kernel Density Construction Using Orthogonal Forward Regression with Leave-One-Out Test Score and Local Regularization

Transcription

1 IEEE TANSACTIONS ON SYSTEMS MAN AND CYBENETICS PAT B: CYBENETICS VOL NO X 1 Sprse Kernel Densit Construction Using Orthogonl Forrd egression ith Leve-One-Out Test Score nd Locl egulriztion S Chen Senior Member IEEE X Hong Senior Member IEEE nd CJ Hrris Abstrct The pper presents n efficient construction lgorithm for obtining sprse kernel densit estimtes bsed on regression pproch tht directl optimizes model generliztion cpbilit Computtionl efficienc of the densit construction is ensured using n orthogonl forrd regression nd the lgorithm incrementll minimizes the leve-one-out test score A locl regulriztion method is incorported nturll into the densit construction process to further enforce sprsit An dditionl dvntge of the proposed lgorithm is tht it is full utomtic nd the user is not required to specif n criterion to terminte the densit construction procedure This is in contrst to n eisting stte-of-rt kernel densit estimtion method using the support vector mchine SVM) here the user is required to specif some criticl lgorithm prmeter Severl emples re included to demonstrte the bilit of the proposed lgorithm to effectivel construct ver sprse kernel densit estimte ith comprble ccurc to tht of the full smple optimized Przen indo densit estimte Our eperimentl results lso demonstrte tht the proposed lgorithm compres fvourbl ith the SVM method in terms of both test ccurc nd sprsit for constructing kernel densit estimtes Inde Terms Cross vlidtion leve-one-out test score orthogonl lest squres Przen indo estimte probbilit densit function sprse kernel modeling regulriztion I INTODUCTION Estimtion of probbilit densit functions is recurrent theme in mchine lerning nd mn fields of engineering see for emple [1] [] A ell-knon non-prmetric densit estimtion technique is the clssicl Przen indo estimte [5] hich is remrkbl simple nd ccurte The prticulr problem ssocited ith the Przen indo estimte hoever is the computtionl cost for testing hich scles directl ith the smple size s the Przen indo estimte emplos the full dt smple set in defining densit estimte for subsequent observtions In tod s dt rich environment this cn be serious problem in prcticl pplictions ecentl the support vector mchine SVM) hs been proposed s promising tool for sprse kernel densit estimtion [][7] The motivtion of the SVM densit estimtion comes from the clim tht the SVM cn effectivel perform function pproimtions in high dimensionl spces from finite dt ith sprse representtions Although this effectiveness hs been demonstrted in regression nd clssifiction problems Mnuscript received December 3; revised Mrch S Chen nd CJ Hrris re ith School of Electronics nd Computer Science Universit of Southmpton Southmpton SO17 1BJ UK X Hong is ith Deprtment of Cbernetics Universit of eding eding G AY UK it is knon tht there re lterntive methods for regression nd clssifiction [][] hich cn provide sprser representtions thn the SVM method Currentl the mchine lerning communit is ctivel engged in the investigtion of the SVM densit estimtion method A recent PhD thesis [1] hs proposed n interesting greed technique for kernel densit estimtion This technique constructs sprse kernel densit estimtes using n orthogonl forrd regression OF) tht incrementll minimizes the trining men squre error MSE) [11] This sprse densit construction lgorithm is computtionll simple nd efficient nd the results given in [1] hve demonstrted the potentil of this method One criticl spect of this method hich is less stisfctor is in hen to terminte the densit construction procedure The minimum descriptive length [1] nd Akike s informtion criterion [13] ere first suggested to help terminte the densit construction process but the empiricl results shoed tht models obtined ere still often oversized At the end mimum model size s imposed in order to void n over-fit model Motivted b the promising result in [1] nd our previous ork on sprse dt modeling [1] [1] e propose n efficient construction lgorithm for sprse kernel densit estimtion using the OF bsed on the leve-oneout LOO) test score nd locl regulriztion Specificll e etend the regression model construction lgorithm [1] to the construction of sprse kernel densit estimtes We ill refer to our proposed lgorithm s the sprse densit construction SDC) lgorithm Our motivtion is tofold Firstl e im to derive sprse kernel densit estimtes bsed on optimizing model generliztion cpbilit or test performnce We lso nt the kernel densit construction process to be utomtic ithout the need for the user to specif some dditionl termintion criterion The usul trining MSE cnnot chieve these objectives but the delete-one cross vlidtion ith its ssocited LOO test score [17] [] provides the cpbilit to chieve this im ithout resorting to use seprte vlidtion dt set Secondl the level of sprsit nd computtionl efficienc re lso criticl to the kernel densit construction process The computtionl efficienc of using the delete-one cross vlidtion is ensured b using the orthogonl lest squres lgorithm [1][] s is first shon in [] nd multipleregulrizers or locl regulriztion is knon to be cpble of providing ver sprse solutions [][1] [1] Our previous ork on sprse regression modeling [1] hs shon tht the OF bsed on the LOO test score nd locl regulriztion

2 C T T M h h W W t J t T IEEE TANSACTIONS ON SYSTEMS MAN AND CYBENETICS PAT B: CYBENETICS VOL NO X offers considerble dvntges in relizing these to criticl objectives of sprse modeling over severl other stte-of-rt methods The current investigtion shos tht the proposed SDC method inherits these crucil dvntges Compred ith the SVM method our SDC lgorithm is simpler to implement nd hs no criticl lgorithm prmeter tht needs to be specified b the user Severl emples re used to illustrte the bilit of this ne SDC lgorithm to construct efficientl sprse densit estimte ith comprble ccurc to tht of the Przen indo estimte Some emples tht hve been used in the eisting literture to investigte the SVM method re specificll chosen in order to compre the performnce of our SDC lgorithm ith the SVM densit estimtion method Our eperimentl results demonstrte tht the SDC lgorithm offers vible lterntive to the SVM method for constructing sprse nd ccurte kernel densit estimtes here W Y ^] is the unknon cumultive distribution function corresponding to the densit ^] Given the dt set the /_?N defined b W empiricl distribution function /_?N@ ith b 'dc ` <b L : ) : %e 3 3f %g 3 ) is knon WY to be good pproimtion to the true distribution function [][7] Thus the kernel densit estimtion problem cn be posed s the folloing regression modeling problem [][7][1]: /_>N@ )< h /?:ikj 1) II KENEL DENSITY ESTIMATION AS EGESSION drn Consider the finite smple set from densit here the dt smples! #%$ re ssumed to be independentl identicll distributed The tsk is to estimte the unknon densit using the kernel densit estimte of the form ith the constrints ' )+ nd ) - / 13577:;< > 7 In this stud the kernel function is ssumed to be the Gussin function of the form /?:@ 1) ) 3) ;BADC FE>FG)H<IKJDLNM L ;BC O ) C here is common kernel idth The ell-knon Przen indo estimte [5] is obtined b setting for ll Our im is to seek spre representtion for ie ith most of being zero nd et mintining comprble test performnce or generliztion cpbilit to tht of the full smple Przen indo estimte hving n optimized vlue for A densit is defined s the solution of the eqution P#Q DS subject to the constrints P S ZS nd UT <V [T <V 13 XW Y \ 5) ) 7) /?: subject to the constrints ) nd 3) here the regressor is given b / l P Q DS ` UT J Lnm fv ith m ' P S p ;-A GH+I J Lsr q ; O V j nd denotes the modeling error t Define t! W:uvW +_?N h zh h! h h ith f}~ model 1) for the dt point u W W i j j here 5 L C OoO 11) r 1) nd Then the regression cn be epressed s ikj 13) Furthermore the regression model 1) over the trining dt set cn be ritten together in the mtri form Xƒ ik 1) ith the gš ƒ hu 'K$ ˆ f folloing > g dditionl nottions j uj ;:Z?j N~! ith nd W:/W W ƒœ! For convenience e ill denote the regression mtri ith h h ' >h! should not ƒ be confused ith 5 the former ƒ is the th column of nd the ltter the th ro of ) Let n orthogonl decomposition of the regression mtri ƒ here nd ƒž š be 15) 1) 17)

3 j j 1 g IEEE TANSACTIONS ON SYSTEMS MAN AND CYBENETICS PAT B: CYBENETICS VOL NO X 3 ith columns stisfing 3 model 1) cn lterntivel be epressed s ik here the eight vector : if The regression 1) ssocited ith the orthogonl spce stisfies the tringulr sstem t The spce spnned b the originl model bses g g is identicl to the spce spnned b W the orthogonl model bses nd the model is equivlentl epressed b W: 1) here ' is ƒ the th ro of In generl the regression mtri in 1) m be illconditioned or even non-invertible prticulrl for lrge dt set This cn cuse numericl problems for some densit construction lgorithms but not the proposed SDC lgorithm This is becuse the OF utomticll voids n ƒ ill-conditioning problems nd selects subset mtri of tht is ellconditioned III THE SPASE DENSITY CONSTUCTION In the OF lgorithm bsed on the LOO test score nd locl regulriztion [1] the eight prmeter vector is the regulrized lest squres solution obtined b minimizing the folloing regulrized error criterion '@ Fi ) here! is the regulriztion prmeter vector hich is optimized bsed on the evidence procedure [3] ith the itertive updting formuls [15][1] here L - i nd v % 1) ) Usull fe itertions tpicll less thn 1) re sufficient to find locl optiml The criterion ) hs its root in the Besin lerning frmeork For the completeness this Besin interprettion together ith the derivtion of the updting formuls 1) nd ) re summrized in Appendi A An OF procedure is used to construct sprse densit estimte b incrementll minimizing the LOO test score Assume tht n -term model is selected from the full model 1) Then the LOO test error [17] [] denoted s j! + for the selected -term model cn be shon to be [][1] here j j is the -term modeling error nd ssocited LOO error eighting given b 5' L 3) i 5 is the ) The men squre LOO error for the model ith size defined b s$#% j + 'o j is 5) This LOO test score cn be computed efficientl due to the fct tht the -term model error j) nd the ssocited LOO error eighting cn be clculted recursivel ccording to L Xj 5 L L i L i ) 7) respectivel For the benefits of those reders ho re unfmilir ith the LOO sttistics the ide of delete-1 cross vlidtion nd the computtion of the LOO test error re eplined in Appendi B The subset model selection procedure cn be crried s follos: t the th stge of the selection procedure model term is selected mong the remining to cndidtes if the resulting -term model produces the smllest LOO test score It hs been shon in [] tht the LOO sttistic is conve ith respect to the model size Tht is there eists n optiml model size such tht i for decreses s increses hile for increses s increses This propert is etremel useful s it enbles the selection procedure to be utomticll terminted ith n -term model hen -/ e + - ithout the need for the user to specif seprte termintion criterion The itertive SDC procedure bsed on this OF ith LOO test score nd locl regulriztion cn no be summrized: g g Initiliztion Set to the sme smll 7 positive vlue eg 1) Set itertion inde Step 1 Given the current nd ith the folloing initil conditions j1 5' WB 1 nd 1 3 '7: g g ) W ) use the procedure described in Appendi C to select subset model ith + terms Step Updte using 1) nd ) ith 5 If remins sufficientl unchnged in to successive itertions or pre-set mimum itertion i number eg 1) is reched stop; otherise set nd go to Step 1 The computtionl compleit of the bove lgorithm is dominted b the 1st itertion After the 1st itertion the model set contins onl 7 terms nd the compleit of the subsequent itertion decreses drmticll As probbilit densit the constrint ) must be met In [1] the non-negtive condition ) is gurnteed b using bckrd elimintion Let be the subset mtri of corresponding

4 C - + IEEE TANSACTIONS ON SYSTEMS MAN AND CYBENETICS PAT B: CYBENETICS VOL NO X 5 5 true pdf Przen true pdf SDC p) 1 p) ) b) Fig 1 ) true densit solid) nd Przen indo estimte dshed) nd b) true densit solid) nd sprse densit construction estimte dshed) for the one-dimensionl emple to the -term model nd nd t the ssocited orthogonl nd originl eight vectors respectivel linked b t If dding the th term cuses some of the elements in t to become negtive the ssocited previousl selected model terms re removed This strteg requires to crr out reorthogonliztion nd in prticulr re-clcultion of the LOO test score hich re computtionll epensive We dopt much simple method to gurntee the non-negtive condition ) In the th stge cndidte tht cuses t to hve negtive elements if included ill not be considered t ll The unit length condition 3) cn esil be met b normlizing the finl + -term model eights ith - gk 'g IV NUMEICAL EXAMPLES + ) Four emples ere used in simultion to test the proposed SDC lgorithm nd to compre its performnce ith the Przen indo estimte Comprison ith SVM kernel densit estimtion s lso given b quoting the C results of [7] In order to remove the influence of different vlues to the qulit of the resulting densit estimte the optiml vlue for found empiricll C b cross vlidtion s used Tht is the vlue of used s determined b testing performnce For the first three emples in ech cse dt set of rndoml drn smples s used to construct kernel densit estimtes nd seprte test dt set of 35?3:3 3 smples s TABLE I PEFOMANCE OF THE PAZEN WINDOW PW) ESTIMATE AND THE POPOSED SPASE DENSITY CONSTUCTION SDC) ALGOITHM FO THE ONE-DIMENSIONAL EXAMPLE STD: STANDAD DEVIATION ) method test error men STD) kernel number men STD) PW SDC! #$% ' used to clculte the ccording to F test error for the resulting estimte +- : L : 3) The eperiment s repeted b 1 different rndom runs for ech emple The fourth emple s to-clss todimensionl clssifiction problem tken from [] Emple 1 This s one-dimensionl emple nd the densit to be estimted s given b l 3 / p ;BA GH<I JL i3 / L ;: ; O 3 1 ; GH+I L 3 1 ik; 31) The number of dt points for densit estimtion s 3 3 C 3 // C The optiml kernel idths ere found to be nd 3 empiricll ith cross vlidtion for the Przen indo estimte nd the SDC estimte respectivel Tble I compres the performnce of the to kernel densit construction methods in terms of the test error nd the number of kernels required Fig 1 ) depicts the Przen indo estimted obtined in run hile Fig 1 b) shos the densit obtined b the SDC lgorithm in run in comprison ith the true distribution For this one-dimensionl emple it cn be seen tht the ccurc of the proposed SDC lgorithm s comprble to tht of the Przen indo estimte nd the lgorithm relized ver sprse estimtes ith n verge kernel number less thn 5% of the dt smples This emple s considered in [7] here SVM Gussin kernel densit estimte of 5 terms s identified ; from single set of 1 trining dt ith n test error of 3/5 over test set of 1 smples It cn be seen tht the result obtined b the SDC method compres fvourbl ith tht of SVM method Emple The densit to be estimted for this to- 3

5 i IEEE TANSACTIONS ON SYSTEMS MAN AND CYBENETICS PAT B: CYBENETICS VOL NO X pdf 3 pdf ) ) Fig True densit ) nd contour plot b) for the to-dimensionl emple b) dimensionl emple s defined b f 3 / ;BA GH+I%JL L ;: ; O G)H<IKJDL L ;: ; O 3 / 3 / GH<I L 3 1 ik; 5 G)H<I L 3 / i ; 3) Fig shos this densit distribution nd /B3 3 its contour plot The estimtion dt set contined smples C nd 3 the empiricll found optiml kernel idths ere CX for the Przen indo estimte nd for the SDC estimte respectivel Tble II lists the test errors nd the numbers of kernels required for the to densit estimtion methods A tpicl Przen indo estimte nd tpicl SDC estimte re depicted in Figs 3 nd respectivel Agin for this emple the to densit construction methods hd comprble ccurcies but the SDC lgorithm chieved ver sprse estimtes ith n verge number of required kernels less thn 3% of the dt smples This emple s lso tken from [7] here SVM Gussin kernel densit estimte of 7 terms s identified from single set of onl trining dt ith n test Fig 3 A Przen indo estimte ) nd contour plot b) for the todimensionl emple b) TABLE II PEFOMANCE OF THE PAZEN WINDOW PW) ESTIMATE AND THE POPOSED SPASE DENSITY CONSTUCTION SDC) ALGOITHM FO THE TWO-DIMENSIONAL EXAMPLE STD: STANDAD DEVIATION THE NUMBE OF TAINING POINTS WAS 5 method test error men STD) kernel number men STD) PW ) SDC - ) ) over test set of 1 smples For comprison e lso performed the eperiments over 1 rndom runs ech ith trining dt points nd the results re listed in Tble III Agin the ccurc of the SDC lgorithm is comprble to tht of the Przen indo estimte Obviousl ith such short trining dt length the stndrd devition of estimte s lrge Inspecting the results of the SDC lgorithm it s found tht 5% of runs ielded kernel densit estimtes of less thn 7 terms ith test errors smller thn / 3:3 35# 3 This gin demonstrtes tht the SDC method compres fvourbl ith SVM method error of / 3:3 5\3

6 c IEEE TANSACTIONS ON SYSTEMS MAN AND CYBENETICS PAT B: CYBENETICS VOL NO X pdf TABLE III PEFOMANCE OF THE PAZEN WINDOW PW) ESTIMATE AND THE POPOSED SPASE DENSITY CONSTUCTION SDC) ALGOITHM FO THE TWO-DIMENSIONAL EXAMPLE STD: STANDAD DEVIATION THE NUMBE OF TAINING POINTS WAS method test error men STD) kernel number men STD) PW ) SDC! 1 TABLE IV PEFOMANCE OF THE PAZEN WINDOW PW) ESTIMATE AND THE 5 5 ) POPOSED SPASE DENSITY CONSTUCTION SDC) ALGOITHM FO THE SIX-DIMENSIONAL EXAMPLE STD: STANDAD DEVIATION method test error men STD) kernel number men STD) PW -! # SDC #) Fig A sprse densit construction estimte ) nd contour plot b) for the to-dimensionl emple Emple 3 In this si-dimensionl emple the underling densit to be estimted s given b ith ' b) E> 5 ;-A G G)H<I%JDL ; L L O i G ' G)H<IKJDL ; L L O i G G)H<I J L ;@ L L O 33) ! o dig 35>; 35 3f; 3f 3f>; 3f 3) L 3 L 3 L 3 L 3 L 3 L 3- ; dig 3f 3f; 3f 35>; ) 3 3s3 ; 3s3 3s3 3s3 3s3 3-! dig 35 35>; 3f 3f; 3f 3f 3) 3:3 3 The estimtion dt set contined smples The optiml kernel idth s found C C 3 to be 3 for the Przen indo estimte nd for the SDC estimte respectivel vi cross vlidtion using the test dt set The results obtined b the to densit construction lgorithms re summrized in Tble IV It cn be seen tht the SDC lgorithm chieved similr ccurc to tht of the Przen indo estimte ith much sprser representtion The verge number of required kernels for the SDC method s less thn 3% of the dt smples Emple The dt s obtined from This s the snthetic dt set tken from [] hich s to-clss clssifiction problem in to-dimensionl feture spce The trining set contined 5 smples ith 15 points for ech clss nd the test set hd 1 points ith 5 smples for ech clss Tipping [] reported tht the optiml Bes error rte for this emple is round % ho lso constructed SVM Gussin kernel clssifier of 3 kernel functions ith test error rte of 1% nd relevnce vector mchine Gussin kernel clssifier of kernel functions ith test error rte of 3% We first estimted the to conditionl C nd densit functions C1 from the trining dt nd then pplied the Bes decision rule if 1 C ) C1 belongs to clss 37) else belongs to clss 1 to the test dt set nd clculted the corresponding error rte Tble V lists the results obtined b the to kernel densit construction methods the Przen indo estimte nd the SDC lgorithm here the vlue of C s determined b minimizing the test error rte It cn be seen tht the SDC method ielded ver sprse conditionl densit estimtes nd the resulting test error s ver close to the optiml Bes clssifiction performnce This clerl demonstrted the ccurc of the densit estimtes This result compres fvourbl ith the results of the stte-of-rt kernel clssifiers reported in [] Fig 5 ) nd b) depict the decision boundries of

7 ` IEEE TANSACTIONS ON SYSTEMS MAN AND CYBENETICS PAT B: CYBENETICS VOL NO X 7 TABLE V PEFOMANCE OF THE PAZEN WINDOW PW) ESTIMATE AND THE POPOSED SPASE DENSITY CONSTUCTION SDC) ALGOITHM FO THE TWO-CLASS CLASSIFICATION POBLEM C kernel idth C1 kernel idth test error rte method PW 15 kernels 15 kernels 1% SDC 5 kernels kernels 3% ) Fig 5 ) decision boundr of the Przen indo estimte nd b) decision boundr of the sprse densit construction estimte here circles represent the clss-1 trining dt nd crosses the clss- trining dt the clssifier 37) for the Przen indo nd SDC methods respectivel V CONCLUSIONS An efficient construction lgorithm hs been presented for obtining kernel densit estimtes bsed on n orthogonl forrd regression procedure tht incrementll minimizes the leve-one-out test score coupled ith locl regulriztion to further enforce the sprseness of densit estimte representtions The proposed method is simple to implement nd computtionll efficient nd ecept for the kernel idth the lgorithm contins no other free prmeters tht require tuning The bilit of the proposed lgorithm to construct ver sprse kernel densit estimte ith comprble ccurc to tht of the full smple Przen indo estimte hs been demonstrted using severl emples The results obtined hve shon tht the proposed method provides vible lterntive to the stteof-rt support vector mchine method for sprse kernel densit estimtion in prcticl pplictions APPENDIX A According to the Besin lerning theor eg [3][]) the optiml is obtined b mimizing the posterior probbilit of hich is given b z ' ' z z 3) z 1 b) here denoting the vector of hperprmeters j nd noise prmeter the inverse of the vrince of ) ' is the likelihood nd z is the evidence tht does j not depend on eplicitl Under the ssumption tht is hite nd hs Gussin distribution the likelihood is epressed s is the prior ith! z ' ;-A E> G)H<I L If the Gussin prior is chosen nmel '? p p ;-A G)H<I J L ; 3) ; O ) mimizing ith respect to is equivlent to minimizing the folloing Besin cost function z ' oi 1) It is esil seen tht the criterion ) is equivlent to the criterion 1) ith the reltionship here dig v g 'g ) The hperprmeters specif the prior distributions of Since initill one does not kno the optiml vlue of should be initilized to the sme smll vlue nd this corresponds to choose sme flt distribution for ech prior of in ) The beut of Besin lerning is let dt

8 i L j! IEEE TANSACTIONS ON SYSTEMS MAN AND CYBENETICS PAT B: CYBENETICS VOL NO X spek it lerns not onl the model prmeters but lso the relted hperprmeters This cn be done for emple b itertivel optimizing nd using n evidence procedure [3][] Folloing McK [3] it cn be shon tht the log model evidence for nd is pproimted s z? ; L ; L ; L ; A ; ; G 3) here is set to the mimum posterior probbilit solution nd the Hessin mtri is digonl nd is given b Setting Setting Note ith i dig /i B i s i Y 3 ˆ L Y 3 nd define ) ields the re-clcultion formul for i 5) ields the re-clcultion formul for i [ ) i Then the re-clcultion formul for L : APPENDIX B 7) i ) is v gk zg ) Consider the model selection problem here set of models hve been identified using the trining dt set W? Denote these models identified using ll the dt points of W s 5 nd the corresponding modeling errors s j 'XWB L W 7 :;< ) 5) ith inde + A commonl used cross vlidtion for model selection is the delete-1 cross vlidtion The ide is s follos For ever model ech dt point in the trining set is sequentill set side in turn model is estimted using the remining L dt points nd the prediction error is derived using onl the dt point tht s removed from trining Specificll let be the resulting dt set b removing the th dt point from nd denote the th model estimted using relted predicted model residul t s W 5 W L W s W nd the 51) The men squre LOO test error [17][1] for the th model is obtined b verging ll these prediction errors: # % j ' 5) The men squre LOO test error is mesure of the model generliztion cpbilit To select the best model from the W cndidte models 5 g g the sme modeling procedure is pplied to ech of the predictors nd the model ith the minimum LOO test error is selected For liner-in-the-eights models the LOO test errors cn be generted ithout ctull sequentill splitting the trining dt set nd repetedl estimting the ssocited models b using the Shermn-Morrison-Woodbur theorem [17] Moreover ithin the OF model selection procedure the LOO test errors for the -term model cn be computed ver efficientl 5 It cn redil be j < shon [][1] tht the computtion of the LOO error for the -term model is bsed on the previousl selected L -term model nd the currentl selected th model term vi the efficient recursion formuls ) nd 7) APPENDIX C The modified Grm-Schmidt orthogonliztion procedure ƒ [1] clcultes the mtri ro b ro nd orthogonlizes s follos: t the th stge mke the columns i# g g orthogonl to the th column nd repet the opertion for g g L 1 Specificll denoting g g then for 7:>;f? L i g g 53) L i g g The lst stge of the procedure is simpl 1 The elements of similr : re computed b trnsforming i L g g in 5) This orthogonliztion scheme cn be used to derive simple nd efficient lgorithm for selecting subset models in forrd-regression mnner [1] First define ƒ $# % 55) If some of the columns ) ƒ in ƒ hve been interchnged this ill still be referred to s for nottionl convenience Let denote the subset mtri of corresponding to the -term model nd nd t the ssocited orthogonl nd originl eight vectors respectivel Let ver smll positive number stisfing t

9 i IEEE TANSACTIONS ON SYSTEMS MAN AND CYBENETICS PAT B: CYBENETICS VOL NO X be given hich specifies the zero threshold nd is used to utomticll voiding n ill-conditioning or singulr problem With the initil conditions s specified in ) the th stge of the selection procedure is given s follos g g Step 1 For : Test 1 Conditioning number check If the th cndidte is not considered Test Non-negtiveness check Compute J O Set nd solve t for t If t contins negtive elements the th cndidte is not considered g g Compute for j 5'XW L W here nd be \ nd g g Step Find L j / re the th elements of respectivel Let the inde set nd psses both Tests 1 nd!# $ Then the ƒ th column of is interchnged ith the ƒ th column of the th column of is interchnged ith the th column of up to the L th ro nd the th element of is interchnged ith the th element of This effectivel selects the th cndidte s the th regressor in the subset model Step 3 The selection procedure is terminted ith L -term model if e Otherise perform the orthogonliztion s indicted in 53) to derive the ƒ ƒ -th ro of nd to trnsform into ; clculte nd updte into in the shon in 5); updte the LOO error eightings 5@ L nd go to Step 1 i! s >;f )? [] S Chen AK Smingn B Mulgre nd L Hnzo Adptive minimum-be liner multiuser detection for DS-CDMA signls in multipth chnnels IEEE Trns Signl Processing Vol No pp [5] E Przen On estimtion of probbilit densit function nd mode The Annls of Mthemticl Sttistics Vol33 pp [] J Weston A Gmmermn MO Stitson V Vpnik V Vovk nd C Wtkins Support vector densit estimtion in: B Schölkopf C Burges nd AJ Smol eds Advnces in Kernel Methods Support Vector Lerning MIT Press Cmbridge MA 1 pp3-3 [7] S Mukherjee nd V Vpnik Support vector method for multivrite densit estimtion Technicl eport AI Memo No 153 MIT AI Lb 1 [] ME Tipping Sprse Besin lerning nd the relevnce vector mchine J Mchine Lerning eserch Vol1 pp11 1 [] SS Chen DL Donoho nd MA Sunders Atomic decomposition b bsis pursuit SIAM evie Vol3 No1 pp [1] A Choudhur Fst Mchine Lerning Algorithms for Lrge Dt PhD Thesis Computtionl Engineering nd Design Center School of Engineering Sciences Universit of Southmpton [11] PB Nir A Choudhur nd AJ Kene Some greed lerning lgorithms for sprse regression nd clssifiction ith Mercer kernels J Mchine Lerning eserch Vol3 pp71 1 [1] MH Hnsen nd B Yu Model selection nd the principle of minimum description length J Americn Sttisticl Assocition Vol No5 pp [13] H Akike A ne look t the sttisticl model identifiction IEEE Trns Automtic Control VolAC-1 pp [1] S Chen Locll regulrised orthogonl lest squres lgorithm for the construction of sprse kernel regression models in Proc th Int Conf Signl Processing Beijing Chin) Aug-3 pp1 13 [15] S Chen X Hong nd CJ Hrris Sprse kernel regression modeling using combined locll regulrized orthogonl lest squres nd D- optimlit eperimentl design IEEE Trns Automtic Control Vol No pp [1] S Chen X Hong CJ Hrris nd PM Shrke Sprse modeling using orthogonl forrd regression ith PESS sttistic nd regulriztion IEEE Trns Sstems Mn nd Cbernetics Prt B to pper [17] H Mers Clssicl nd Modern egression ith Applictions nd Edition Boston: PWS-KENT 1 [1] LK Hnsen nd J Lrsen Liner unlerning for cross-vlidtion Advnces in Computtionl Mthemtics Vol5 pp 1 [1] G Monri nd G Drefus Locl overfitting control vi leverges Neurl Computtion Vol1 pp11 15 [] X Hong PM Shrke nd K Wrick Automtic nonliner predictive model construction lgorithm using forrd regression nd the PESS sttistic IEE Proc Control Theor nd Applictions Vol15 No3 pp5 5 3 [1] S Chen SA Billings nd W Luo Orthogonl lest squres methods nd their ppliction to non-liner sstem identifiction Int J Control Vol5 No5 pp [] S Chen CFN Con nd PM Grnt Orthogonl lest squres lerning lgorithm for rdil bsis function netorks IEEE Trns Neurl Netorks Vol No pp [3] DJC McK Besin interpoltion Neurl Computtion Vol No3 pp [] BD iple Pttern ecognition nd Neurl Netorks Cmbridge: Cmbridge Universit Press 1 EFEENCES [1] CM Bishop Neurl Netorks for Pttern ecognition Oford UK: Oford Universit Press 15 [] BW Silvermn Densit Estimtion London: Chpmn Hll 1 [3] H Wng obust control of the output probbilit densit functions for multivrible stochstic sstems ith gurnteed stbilit IEEE Trns Automtic Control Vol No11 pp

10 IEEE TANSACTIONS ON SYSTEMS MAN AND CYBENETICS PAT B: CYBENETICS VOL NO X 1 PLACE PHOTO HEE Sheng Chen SM 7) received the BEng degree in control engineering from the Est Chin Petroleum Institute Donging Chin in 1 nd the PhD degree in control engineering from the Cit Universit London UK in 1 He joined the Deprtment of Electronics nd Computer Science Universit of Southmpton Southmpton UK in September 1 He previousl held reserch nd cdemic ppointments t the Universit of Sheffield Sheffield UK the Universit of Edinburgh Edinburgh UK nd Universit of Portsmouth Portsmouth UK His recent reserch orks include dptive nonliner signl processing modeling nd identifiction of nonliner sstems neurl netork reserch finite-precision digitl controller design evolutionr computtion methods nd optimiztion He hs published over reserch ppers PLACE PHOTO HEE Xi Hong SM ) received the BSc nd MSc degrees from Ntionl Universit of Defense Technolog Chngsh Chin in 1 nd 17 respectivel nd the PhD degree from the Universit of Sheffield Sheffield UK in 1 ll in utomtic control She orked s reserch ssistnt in the Beijing Institute of Sstems Engineering Beijing Chin from She orked s reserch fello in the Deprtment of Electronics nd Computer Science Universit of Southmpton Southmpton UK from 17-1 She is currentl lecturer t the Deprtment of Cbernetics Universit of eding eding UK She is ctivel engged in reserch into neurofuzz sstems dt modeling nd lerning theor nd their pplictions Her reserch interests include sstem identifiction estimtion neurl netorks intelligent dt modeling nd control She hs published over 3 reserch ppers nd co-uthored reserch book Dr Hong received Donld Julius Groen Prize from IMechE UK in 1 PLACE PHOTO HEE Chris Hrris receiving the BSc degree from the Universit of Leicester Leicester UK the MA degree from the Universit of Oford Oford UK nd the PhD degree from the Universit of Southmpton Southmpton UK He previousl held ppointments t the Universit of Hull Hull UK the UMIST Mnchester UK the Universit of Oford Oford UK nd the Universit of Crnfield Crnfield UK s ell s being emploed b the UK Ministr of Defense He returned to the Universit of Southmpton s the Lucs Professor of Aerospce Sstems Engineering in 17 to estblish the Advnced Sstems eserch Group nd more recentl ISIS His reserch interests lie in the generl re of intelligent nd dptive sstems theor nd its ppliction to intelligent utonomous sstems such s utonomous vehicles mngement infrstructures such s commnd control intelligent control nd estimtion of dnmic processes multi-sensor dt fusion nd sstems integrtion He hs uthored nd co-uthored 1 reserch books nd over 3 reserch ppers nd he is the ssocite editor of numerous interntionl journls including Dr Hrris s elected to the ol Acdem of Engineering in 1 s rded the IEE Senior Achievement medl in 1 for his ork in utonomous sstems nd the highest interntionl rd in IEE the IEE Frd medl in 1 for his ork in intelligent control nd neurofuzz sstems