Parameters Tuning in Support Vector Regression for Reliability Forecasting

Size: px

Start display at page:

Download "Parameters Tuning in Support Vector Regression for Reliability Forecasting"

Chester Lindsey
6 years ago
Views:

1 A publcato of CHEMICAL ENGINEERING RANSACIONS VOL. 33, 03 Guest Edtors: Erco Zo, Pero Barald Copyrght 03, AIDIC Servz S.r.l., ISBN ; ISSN he Itala Assocato of Chemcal Egeerg Ole at: DOI: /CE Parameters ug Support Vector Regresso for Relablty Forecastg We Zhao a,ao ao a, Erco Zo b, c a Group 03, School of Electroc ad Iformato Egeerg, Behag Uversty, Bejg, 009, Cha b Char o Systems Scece ad the Eergetc Challege, Europea Foudato for New Eergy-Electrcté de Frace, Ecole Cetrale Pars ad Supelec, Pars, Frace c Dpartmeto d Eerga, Poltecco d Mlao, Mlao, Italy zhaowe03@buaa.edu.c he recet ad promsg mache learg techque called support vector mache (SVM) has become a hot research subject tme seres forecastg, sce proposed from Statstc Learg heory by Vapk. As a mportat applcato of tme seres forecastg, relablty predcto by aalyzg the hstorcal tme seres data of system codto to predct the future system behavour ad/or dagose the possble system fault, has bee solved successfully by SVM wth hgh forecastg accuracy. For ths, the crtcal problem s the selecto of SVM parameters. May methods have bee proposed, such as geetc algorthm, partcle swarm optmzato ad aalytc selecto; but there s o geerally structured way, yet. I ths paper, the capablty of SVM to perform fucto fttg ad relablty forecastg based o dfferet methods s vestgated by expermetg o both artfcal ad real-world data. A comparso of the methods s offered o crtera of predcto accuracy ad robustess. Fally, a attempt s made to obta a comparatve optmal parameter selecto method.. Itroducto Safe ad relable operato of egeerg systems s very mportat. o guaratee ths, relablty aalyss ad rsk assessmet offer soud techcal frameworks for the study of compoet ad system falures, wth quatfcato of ther probabltes ad cosequeces (Zo, 009). I ths frameworks, oe mportat goal s relablty predcto. Uder certa codtos, relablty predcto ca be see as a tme seres predcto problem whose soluto etals predctg the future values of relablty based o past data observatos. A wdely used predcto approach s the ARIMA model, wth sold foudatos classcal probablty theory. However, the tme-cosumg off-le modellg efforts requred for model detfcato ad buldg lmts ts usefuless practcal applcatos (Lu et al., 00). I recet years, eural etwork has emerged as a uversal approxmator for ay olear cotuous fucto varyg over a tme or space doma, ad has bee appled successfully to varous relablty problems such as software relablty predcto (Ada ad Yaacob, 994) ad complex system mateace (Amjady ad Ehsa, 999). However, practcal dffcultes are ecoutered due to the eed of large datasets for trag, o guaratee of covergece to optmalty ad the dager of over-fttg (Che, 007, Sapakevych ad Sakar, 009). Aother powerful mache learg paradgm s the Support Vector Mache (SVM) developed by Vapk ad others 995 (Vapk, 995), based o statstcs learg theory ad VC theory. SVM embodes the dea of mmzg the Structure Rsk Mmzato (SRM) rather tha the Emprcal Rsk Mmzato (ERM) adopted eural etwork trag. Sce the ERM prcple s most suted for large trag datasets, SVM has bee prove to provde superor performaces tha eural etworks o small datasets. For ths reaso, SVM has bee appled to may mache learg tasks cludg tme seres predcto ad relablty forecastg. For example, Hog appled the SVM method to predct ege relablty ad compared the predctg performace wth the Duae model, ARIMA model ad geeral regresso eural etworks (Hog ad Pa, 006). Expermet results show that the SVM model has better performace over the other models. Please cte ths artcle as: Zhao W., ao., Zo E., 03, Parameters tug support vector regresso for relablty forecastg, Chemcal Egeerg rasactos, 33, DOI: /CE

2 Whe applyg SVM to regresso ad predcto problems, the performace depeds heavly o the settg of the free meta-parameters of SVM. he, how to select the parameters s a ma ssue for practtoers tryg to apply SVM. he grd searchg algorthms combed wth k-fold cross valdato are ofte used to fd the best value set of the parameters. But the computatoal burde ca be heavy, whch reders ths exhaustve method lttle practcal. A smple but practcal aalytcal selecto approach (AS) ca provde the basc form of the parameters (Cherkassky ad Ma, 004); advaced optmzato algorthms such as smulated aealg (SA) (Pa ad Hog, 006), geetc algorthm (GA) (Che, 007) ad partcle swarm optmzato (PSO) (Ls et al., 0) have also bee used for SVM parameters tug. I ths paper, we vestgate the capablty of SVM parameters tug by AS, GA ad PSO for fucto regresso ad relablty predcto. he vestgato s carred out by way of some expermets o both artfcal ad real world data. he remader of the paper s orgazed as follows. Secto troduces backgroud kowledge about SVR ad the basc theory of AS, GA ad PSO s preseted Secto 3. Secto 4 presets the expermets o artfcal ad real-world datasets through whch the regresso performaces of the three methods are compared. Secto 5 provdes some dscussos ad coclusos o the expermet results.. Support vector maches for regresso Gve a dataset {(, } l D = s y, where s R deotes the l-dmeso put vector, y deotes the realvalued output ad s the umber of data patters,, we cosder, frst, a SVM to estmate the lear regresso fucto: f ( s ) = w s + b () where w ad b are respectvely the weght vector ad tercept of the model that oe eeds to fd for optmal fttg of the data D. l I the olear case, by a olear mappg Φ : R F, where F s the feature space of Φ, the SVM trasforms the complex olear regresso problem to the comparatvely smple problem of fdg the flattest fucto the feature space F (Che, 007). he, the regresso fucto takes a geeral form sutable for both lear ad olear cases: f ( s ) = w Φ ( s ) + b () he, we troduce the ε -sestve loss fucto (Vapk, 995): l= y f( ) 0, y f( s ) ε s ε = (3) y f( s) ε, ο therwse whch gores the error f the dfferece betwee the predcto value obtaed by Eq.() ad the real value s smaller tha, whch s a parameter to be tued. For the error larger tha ε, slack varables ξξ, are troduced to respectvely represet the fuctoal dstace of two possble but mutually exclusve samples. By troducg the ε -sestve loss fucto, we ca measure the emprcal error ad set up a procedure for mmzg t. Besdes, SVM we must also mmze the Eucldea orm of the lear weght w, w, whch s related to the geeralsato ablty of the SVM model traed. he, a compromsed optmal quadratc optmzato problem to detfy the regresso model arses as follows: m J( w, ξξ, ) = w + C ( ξ+ ξ) w, ξξ, = y w Φ( s) b ε + ξ st.. w Φ ( s) + b y ε + ξ ξ, ξ 0 =,..., (4) where C deotes the pealty coeffcet that modulates the trade-off betwee emprcal ad geeralzato errors, ad must be also tued by the aalyst. he soluto of ths quadratc optmzato problem obtaed by the Lagraga dual method gves the optmal w ad b through whch we ca estmate the predcto value umercally: 54

3 f () s = wφ () s + b= α K(, s s ) + b K( s, s ) =Φ( s ) Φ( s ) j j = (5) where K ( s, s ) s the kerel fucto satsfyg the Mercer codto (Boser et al., 99). If ot metoed j specfcally, the kerel fucto used ths paper s the radal bass fucto wth wdth γ also to be tued by the aalyst. 3. Parameter selecto methods 3. AS method he aalytc selecto (AS) method chooses the parameter trplet, X = [ C, εγ, ], drectly from the trag data ad (estmated) ose level aalytcally as follows (Cherkassky ad Ma, 004): I C = max( y + 3 σ y, y 3 σ y ), ε = 3 σ, γ (0. 0.5) rage( s) (6) where y ad σ are the mea ad the stadard devato of the y values, rage () s = max() s m() s, y σ s the estmated ose level of the trag data obtaed by the followg prescrpto va the k- earest-eghbour s method: /5 k σ = ( y ) /5 y (7) k = where y s the regresso value va k-earest-eghbour s method. 3. GA method Geetc algorthms (GA) are a famly of evolutoary computatoal models spred by the theory of evoluto. hese algorthms ecode each potetal soluto of the optmzato problem a smple chromosome-lke data structure, ad the sft the crtcal formato va some recombato operators that mtate bologcal evoluto processes such as survval of the fttest, crossover ad mutato (Whtley, 994). he basc procedure of GA method adopted our work s descrbed as follows (Che, 007): ) Represetato: Chromosome X s drectly represeted as a SVM parameter vector X = [ C, ε, γ ]. ) Ftess: he ftess value evaluatg the qualty of chromosome X s defed as the mea square error of the 5-fold cross valdato ( MSE CV ) method o the trag data wth SVM parameters X. 3) Italzato ad selecto: I ths study, the tal populato s composed of 40 chromosomes radomly geerated wth the gve rages ov varablty of the three parameters to be tued ad the stadard roulette wheel method s employed to select survval chromosomes from the curret populato, proporto to ther ftess values. 4) Crossover ad mutato: As the core operato of GA, crossover ad mutato play a fudametal role the progress of searchg the best chromosome. I our study, the smulated bary crossover ad polyomal mutato methods are chose to realse the accordg operatos. he probablty of crossover p c ad of mutato pm are respectvely set to 0.8 ad ) Eltst strategy: he chromosome wth the best ftess wll skp the crossover ad mutato procedure ad drectly survve utl the ext geerato. 6) Stoppg crtera: steps 3-5 are repeated for a predefed umber of geeratos ( our applcato ths s set to 00). 3.3 PSO method Partcle swarm optmzato (PSO) s a populato-based meta-heurstcs that smulates socal behavour such as brds flockg to a promsg posto (L et al., 008). PSO performs searches through a populato (called swarm) of dvdual solutos (called partcles) that update teratvely. Each partcle at terato t ca be represeted by a D-dmesoal state vector as X t { t, t,..., t = X X XD}. he, to obta the optmal soluto, we defe D-dmesoal velocty vectors V t { t, t,..., t = V V VD} for each partcle ad determed by ts ow best prevous experece ( pbest ) ad the best experece of all other partcles 55

4 ( gbest ). Partcles chage velocty accordg to the pbest ad gbest as follows: V = V + cr( pbest X ) + cr( gbest X ), d =,,..., D (8) t t t t t t d d d d d d where c, c are the learg factors set to ths study ad r, r are radom umbers dstrbuted uformly the rage (0, ),.e. U(0,). he, each partcle updates to a ew potetal soluto based o the velocty as: t t t X + = X + V, =,,..., (9) d d d d D Whe the terato umber reaches a pre-determed maxmum terato umber, the update process s termated ad the best dvdual of the last geerato s the fal soluto to the target problem. 4. Expermets results I ths Secto, we perform some smulated expermets to vestgate the capablty of these three methods for optmal searchg the SVM parameters. We cosder fucto regresso problems whch are ot drectly related to the relablty predcto problem of terest but hold smlar characterstcs whle, o the other had, beg easly mplemeted ad cotrollable. hrough these regresso cases, we ca systematcally compare the predcto performace of the three methods for optmal SVM parameter detfcato terms of accuracy, stablty ad sestvty to ose. he fdgs of these expermet studes wll gude the choces of the settgs of the algorthms for the relablty predcto case of terest. 4. Fucto regresso Frst, we cosder the sc fucto f () s (Borwe et al., 00) f ( s) = 0s( s) / s s [ 0,0] (0) he smulated trag data are pars ( s, y), ( =,..., ), where s are radom uformly sampled the pre-defed rage ad y are geerated as y = f () s + σ. We frst cosder the case wth ose level σ =,ad =40. he test data are also radom-uformly sampled the same rage as the trag data. Fgure, vsually shows that all the three parameter selecto methods are capable of approxmatg the target fucto. GA ad PSO methods yeld better geeralsato performace, at the cost of a much heaver computatoal burde tha the smpler AS method. o compare a tegrated maer the three methods of SVM parameters tug, we evaluate the predcto rsk, defed as the mea squared error (MSE) betwee the SVM estmates ad the correspodg true values of the target fucto output for the test put values. For ths, ad to accout for the radomess of the estmato process, we perform the regresso seve tmes for a same target fucto value. Fgure cofrms the overall superorty of GA ad PSO. Oe ca also otce the fluctuatos GA performace, whch has worse stablty tha the PSO method whch cosstetly gves a hgh predcto accuracy. able gves the results of expermets for dfferet target fucto types ad ose levels. I geeral the PSO ad GA methods perform better tha the AS method. Further, the mea value ad stadard devato of GA method ted to become large as the ose level creases. hs shows the GA method s stablty ad sestvty to ose. O the cotrary, for all fucto types ad ose level cosdered, PSO method performs satsfactorly both mea value ad stadard devato, whch meas a superorty of PSO method both geeralsato performace ad stablty. Outputs trag data target fucto AS-SVR predcto GA-SVR predcto PSO-SVR predcto Iputs Fgure : Comparso of SVM estmates for the case of the sc fucto wth σ = 56

5 Predcto accuracy.5.5 MSE.AS MSE.GA MSE.PSO able : MSE for dfferet fucto types wth dfferet ose levels Nose level σ = σ = σ = 5 σ = 0 σ = 0.5 σ = Expermets tmes Fgure : Estmate MSE for seve tmes for sc fucto wth σ = arget fucto : y= s Methods MSE Mea Stadard varace AS GA PSO AS GA PSO arget fucto : y = s + s+ AS GA PSO AS GA PSO arget fucto : y = s( s) AS GA PSO AS GA PSO Relablty predcto I ths Secto, a relablty predcto expermet cocerg submare falure data s carred out. he data set cotas 70 submare falure tmes that crease approxmately learly as tme goes by except 8 6 Falure tme trag data test data AS-SVR predcto GA-SVR predcto PSO-SVR predcto Samples dex Fgure 3: Relablty results for submare falure data usg AS, GA ad PSO method. 57

6 able : Estmate MSE for relablty predctos secto 4. Methods MSE Mea Stadard devato AS GA PSO for a hoppg correspodece of tme dex 64. Predcto s doe by a oe-step ahead strategy for predctg the ext ((t+)-th) falure tme based o the curret (t-th) falure tme I ths expermet, the frst 60 tme-to-falure data are used as trag set ad the fal 0 data as test set. Because t s dffcult to get good estmates of the ose the trag data practcal relablty predcto applcatos, the AS method, whch reles heavly o the ose level estmates, shows bad performace trackg the tred of t he relablty data. Istead, as Fgure 3 shows, the GA ad PSO methods are both capable of capturg the tred of the falure data. Eve for the hoppg data, PSO provdes a satsfactory predcto performace, whereas GA gves poorer predctos because of weaker geeralzato ablty. I relablty predcto case, he formato reported able cofrm the stablty of the GA method ad superorty of the PSO method. 5. Cocluso I ths work, we have vestgated the AS, GA ad PSO parameter methods for selectg the parameters of SVM regresso ad predcto tasks. Our expermets results suggest that PSO gves superor performaces, whereas AS gves comparatvely low accuracy ad GA s somewhat ustable. Although the performace of AS s ot comparatvely satsfactory, ts extremely low computatoal burde makes t attractve for talzg the parameter values for the GA ad PSO methods ad optmzg ther search evoluto process to accelerate t ad stablze t: how to embed ths to a dyamc ole method s a future research ssue. Referece Ada W.,Yaacob M. A tegrated eural-fuzzy system of software relablty predcto. Software estg, Relablty ad Qualty Assurace, 994. Coferece Proceedgs., Frst Iteratoal Coferece o, - Dec IEEE, Amjady N.,Ehsa M., 999, Evaluato of power systems relablty by a artfcal eural etwork. Power Systems, IEEE rasactos o, 4, Borwe D., Borwe J.M.,Leoard I.E., 00, L p Norms ad the Sc Fucto. he Amerca Mathematcal Mothly, 7, Boser B.E., Guyo I.M.,Vapk V.N. A trag algorthm for optmal marg classfers. Proceedgs of the ffth aual workshop o Computatoal learg theory, 99. ACM, Che K.Y., 007, Forecastg systems relablty based o support vector regresso wth geetc algorthms. Relablty Egeerg & System Safety, 9, Cherkassky V.,Ma Y., 004, Practcal selecto of SVM parameters ad ose estmato for SVM regresso. Neural etworks, 7, 3-6. Hog W.C.,Pa P.F., 006, Predctg ege relablty by support vector maches. he Iteratoal Joural of Advaced Maufacturg echology, 8, L S.W., Yg K.C., Che S.C.,Lee Z.J., 008, Partcle swarm optmzato for parameter determato ad feature selecto of support vector maches. Expert Systems wth Applcatos, 35, Ls I.D., Moura M.C., Zo E.,Droguett E.L., 0, A partcle swarm-optmzed support vector mache for relablty predcto. Qualty ad Relablty Egeerg Iteratoal, 8, Lu H., Kolark W.J.,Lu S.S., 00, Real-tme performace relablty predcto. Relablty, IEEE rasactos o, 50, Pa P.F.,Hog W.C., 006, Software relablty forecastg by support vector maches wth smulated aealg algorthms. Joural of Systems ad Software, 79, Sapakevych N.,Sakar R., 009, me seres predcto usg support vector maches: a survey. Computatoal Itellgece Magaze, IEEE, 4, Vapk V he ature of statstcal learg theory, sprger-verlag New York Ic, New York, USA. Whtley D., 994, A geetc algorthm tutoral. Statstcs ad computg, 4, Zo E., 009, Relablty egeerg: Old problems ad ew challeges. Relablty Egeerg & System Safety, 94,

An Introduction to. Support Vector Machine

An Introduction to. Support Vector Machine A Itroducto to Support Vector Mache Support Vector Mache (SVM) A classfer derved from statstcal learg theory by Vapk, et al. 99 SVM became famous whe, usg mages as put, t gave accuracy comparable to eural-etwork