Na Zhag, Keea Leatham Feature Selecto Based o SVM Photo-Thermal Ifrared (IR) Imagg Spectroscopy Classfcato Wth Lmted Trag Samples NIAN ZHANG ad KEENAN LEATHAM Departmet of Electrcal ad Computer Egeerg, Uversty of the Dstrct of Columba 400 Coectcut Aveue, NW, Washgto, D.C. 0008 USA zhag@udc.edu, keea.leatham@udc.edu Abstract: - I ths paper, we propose a kerel based SVM algorthm wth varable models to adapt to the hgh-dmesoal but relatvely small samples for remote explosve detecto o photo-thermal frared magg spectroscopy (PT-IRIS) classfcato. The algorthms of the represetatve lear ad olear SVM are preseted. The respose plot, predcted vs. actual plot, ad resduals plot of the lear, quadratc, ad coarse Gaussa SVM are demostrated. A comprehesve comparso of Lear SVM, Quadratc SVM, Cubc SVM, Fe Gaussa SVM, Meda Gaussa SVM, Coarse Gaussa SVM s performed terms of root mea square error, R-squared, mea squared error, ad mea absolute error. The excellet expermetal results demostrated that the kerel based SVM models provde a promsg soluto to hgh-dmesoal data sets wth lmted trag samples. Key-Words: - Feature selecto, Support vector mache, SVM, Hgh-dmesoal, Classfcato, Photothermal frared magg spectroscopy 1 Itroducto Recet advaces moder techologes, such as photo-thermal frared (IR) magg spectroscopy techology the applcato of remote explosve detecto, 4D CT-scas techology, ad DNA mcroarrays have produced umerous massve ad mbalaced data. The eeds of classfcato ubqutously exst real-world data-tesve applcatos, ragg from cvla applcatos such as cacer dagoses ad outler detecto stock market tme seres, to homelad securty or defese related applcatos such as remote explosve detecto, llegal drug detecto, ad abormal behavor recogto. I the stuato whe the dmesoalty of data s hgh but wth few data, feature selecto usually becomes mperatve to the learg algorthms because hgh-dmesoal data teds to egatvely affect the effcecy of most learg algorthms. Feature selecto s a effcet dmesoalty reducto techque that selects a optmal subset of the orgal features that provde the best predctve power modelg the data. They are the most dstct features that ca be used to dfferetate samples to dfferet classes. There are a large umber of state-of-the-art feature selecto methods. A smultaeous spectral-spatal feature selecto ad extracto algorthm was proposed for hyperspectral mages spectral-spatal feature represetato ad classfcato. However, t lacks of kerel verso ad thus ts performace o complex datasets s ukow [1]. A regularzed regresso based feature selecto classfer was modfed to a cost-sestve classfer by geeratg ad assgg dfferet costs to each class. Features wll be selected accordg to the classfer wth optmal F-measure order to solve the class mbalace problem []. A feature selecto algorthm usg AdaBoost was preseted to deal wth Haar-lke features for vehcle detecto. The ormalzed feature vector set s used to tra the RBF-SVM classfer wth cross-valdato to select the optmal parameters [3]. A support vector mache (SVM) was appled as a classfer to detfy resdual fuctoal abormaltes athletes sufferg from cocusso usg a multchael EEG data set. The total accuracy of the classfer usg the 10 features was 77.1% [4]. A multple stace learg (MIL) was adopted to descrbe dfferet kds of actos from complexty data sources E-ISSN: 4-3488 85 Volume 13, 017
Na Zhag, Keea Leatham ad preset a boosted exemplar learg (BEL) method to lear the smlarty metrc ad select some represetatve exemplars from the Web for acto recogto. It takes about 98 ms to tra a multple stace SVM (m-svm) for oe exemplar. The proposed m-svm has much better result 65.37% tha the 61.53% usg SVM classfer [5]. Dfferet from the above SVM based learg algorthm, we propose a kerel based SVM algorthm wth varable models to adapt to the hgh-dmesoal but relatvely small samples for remote explosve detecto o photo-thermal frared magg spectroscopy (PT-IRIS) classfcato. The rest of the paper s orgazed as follows. I Secto, the SVM models cludg the lear SVM, quadratc SVM, ad coarse Gaussa SVM are dscussed. I Secto 3, photo-thermal frared magg spectroscopy (PT-IRIS) data set s troduced. I Secto 4, the classfcato performace of lear, quadratc, ad coarse Gausa SVM are demostrated. I addto, a comparso of Lear SVM, Quadratc SVM, Cubc SVM, Fe Gaussa SVM, Meda Gaussa SVM, Coarse Gaussa SVM s preseted terms of varous model statstcs. Fally, the paper s cocluded Secto 5. Types of SVM Algorthms Support Vector Mache (SVM) learg algorthms has bee a actve research topc wth the computer tellgece commuty. Support vector mache (SVM) aalyss s a popular mache learg tool for classfcato ad regresso, frst detfed by Vladmr Vapk ad hs colleagues 199 [6]. SVM regresso s cosdered a oparametrc techque because t reles o kerel fuctos. Support Vector Mache algorthms are utlzed may real world applcatos such as Face Detecto, Text Categorzato, ad Boformatcs. Support Vector Mache algorthm (SVM) s a supervsed mache learg algorthm, whch ca be used for ether classfcato or regresso challeges. The dfferet types of SVM learg algorthms are Lear SVM, Quadratc SVM, ad Cubc SVM. Each Support Vector Mache Algorthm has ther advatages terms of provdg solutos o a data set. For each algorthm we wll be (1) Trag a data set wth Lear SVM, Quadratc SVM, ad Cubc SVM, () Plottg the behavor of each algorthm fgurg out the RSME, R- Squared Value, MSE, MAE, Predcto Speed, Trag Tme, ad (3) Aalyzg the results of each Support Mache Algorthm to see the smlartes ad dffereces of the data. The purpose of these trals s to see f we ca fd some terestg behavors, so we ca fd dfferet methods to optmze SVM algorthms. Show below are the dfferet behavors of each SVM..1 Lear SVM Lear SVM s the ewest fast mache learg data mg algorthm for solvg multclass classfcato problems from ultra large data sets; that mplemets a orgal propretary verso of a cuttg plae algorthm for desgg a lear support vector mache. Lear SVM s a learly scalable route meag that t creates a SVM model a CPU tme, whch scales the sze of the trag data set learly. Our comparsos wth other kow SVM models clearly show ts performace s hghly accurate, mplemeted wth large data sets. It s deal for a Lear SVM to be utlzed cotemporary applcatos such as dgtal advertsemet-commerce, web page categorzato, text classfcato, boformatcs, proteomcs, bakg servces ad may other areas. It provdes solutos of multclass classfcato problems wth ay umber of classes wth hgh dmesoal data both sparse ad dese formats. There s o eed for expesve computg resources other tha a stadard platform whle mplemetg ths algorthm. The algorthm of the Lear SVM s llustrated as follows. Algorthm of the Lear SVM Iput: 1. A trag data set of the form: ( x1, y1),,( x, y). A y that s ether 1 or -1. Procedure: E-ISSN: 4-3488 86 Volume 13, 017
Na Zhag, Keea Leatham 1. Let the gve trag data set of pots be the form of: x, y ),,( x, y ) ( 1 1. Each x s a dmesoal vector. 3. We wat to fd the maxmum marg hyperplae that groups pot of x 1 to the group of pots where y =1. 4. The pots of x have to be satsfed by: ( w x b) = 0 Where w s the ormal vector to the hyperplae.. Quadratc SVM If data sets are ot learly separable, Quadratc SVM s utlzed to pck out a terval betwee two classes. To solve ths problem the data s mapped o to a hgher dmesoal space ad the uses a lear classfer the hgher dmesoal space. For example, a lear separator ca easly classfy the data f we use a quadratc fucto to map the data to two dmesos. The geeral dea s to map the orgal feature space to a hgher-dmesoal feature space where the trag set s separable. As the expaso creases th degrees t allows the data set to be traed a effcet maer. The algorthm of the quadratc SVM s llustrated as follows. Algorthm of the Quadratc SVM Iput: 1. A trag data set of the form: ( x1, y1),,( x, y). A kerel mappg: K( x, = ϕ ( x), ϕ( Procedure: 1. Let the gve trag data set of pots be the form of: ( x 1, y 1),,( x, y ). The polyomal kerel s defed as: K( x, = ϕ( x), ϕ( where c > 0 3. For polyomals the kerel s defed by: T d K ( x, = ( x y + c) d = 4. Usg the multomal theorem the expaso becomes: K( x, = ( + + = = 1 1 j= ( ( = 1 x c x )( x j )( x y + c) = = 1 y c y ) + c ( x y y.3 Gaussa SVM The Gaussa kerel oly depeds o the Eucldea dstace betwee x ad x, ad s based o the assumpto that smlar pots are close oe to each other the feature space ( terms of Eucldea dstace). The algorthm of the cubc SVM s llustrated as follows. Algorthm of the Gaussa SVM Iput: 1. A trag data set of the form: ( x1, y1),,( x, y). A kerel mappg: K( x, = ϕ ( x), ϕ( Procedure: 1. Let the gve trag data set of pots be the form of: ( x1, y1),,( x, y). The Gaussa kerel s defed as: x y K( x, = e γ for a gve parameter γ > 0 3 Photo-Thermal Ifrared (IR) Imagg Spectroscopy (PT-IRIS) Data Set A photo-thermal frared magg spectroscopy (PT-IRIS) techque has recetly bee developed by the Naval Research Laboratory (NRL), Washgto, DC wth uprecedeted spatal resoluto at ~1 mcro [8]. I ths data set, the mxg Ifrared IR absorpto/emsso features causes some complcated ad overlappg samples, whch leads to grad challeges to mult-class classfcato. Specfcally, frared quatum cascade lasers are used to llumate a surface potetally scattered wth samples of terest. If the wavelegth of the thermal emsso of lght s resoat wth collecto features of surface j ) ) E-ISSN: 4-3488 87 Volume 13, 017
Na Zhag, Keea Leatham samples, the sample heats by ~1oC. By varyg the cdet wavelegth, ay samples of terest could be maged [9]. The feature of the PT-IRIS sgal s the temperature crease ormalzed to the average power of the laser pulse at the ed of the laser pulse,.e. Tmax. Tmax as a fucto of exctato ad collecto wavelegth are bult to a feature vector [10], as show Fg. 1. Fg. 1 Data matrx wth feature vectors. Each square s a feature value, Tmax, whch s a fucto of exctato ad collecto wavelegth. Smulated samples clude 4 dfferet partcle dameters (5, 3, ad 7 mm) ad 4 aalytes (RDX, TNT, AN, Sucrose) o 4 substrates (whte pat, steel, glass, polyethylee) usg 38 exctato wavelegths ad 33 collecto wavelegths. Thus, wth 38 exctato wavelegths ad 33 collecto wavelegths, they would geerate 154 features (predctor varables). Therefore, each colum cotas 154 features ad there are oly 13 samples. We may demostrate the sgal matrx for all the 13 samples. Ths ca be see by dsplay the data set false color plot whch wll show vsble or o-vsble parts of the electromagetc spectrum. The false color plot of the data set s show Fg.. The color of the data pot s proportoal to sgal stregth,.e. red represets hgh, ad blue represets low. 4 Smulato Aalyss 4.1 Explore Data ad Results Respose Plot After a regresso model s traed, the regresso model results ca be dsplayed by the respose plot,.e. the predcted respose versus record umber. Holdout or cross-valdato s used, thus each predcto s obtaed usg a model that was traed wthout usg the correspodg observato. Therefore, these predctos are the predctos o the held-out observatos. 80% of the data s used to tra the etwork ad the remag 0% data pots are used as the testg data. The respose plot of lear SVM, quadratc SVM, ad coarse Gaussa SVM are show Fg. 3, Fg. 4, ad Fg. 5, respectvely. Fg. 3 The respose plot of lear SVM. Fg. 4 The respose plot of quadratc SVM. Fg. False color plot of data set. The data set has 13 samples ad 154 features. E-ISSN: 4-3488 88 Volume 13, 017
Na Zhag, Keea Leatham Fg. 5 The respose plot of coarse Gaussa SVM. 4. Plot Predcted vs. Actual Respose The Predcted vs. Actual plot s used to check model performace after trag a model. Use ths plot to uderstad how well the regresso model makes predctos for dfferet respose values. Whe the plot s ope, the predcted respose of our model s plotted agast the actual, true respose. A perfect regresso model has a predcted respose equal to the true respose, so all the pots le o a dagoal le. The vertcal dstace from the le to ay pot s the error of the predcto for that pot. A good model has small errors, ad so the predctos are scattered ear the le. Usually a good model has pots scattered roughly symmetrcally aroud the dagoal le. If we ca see ay clear patters the plot, t s lkely that we ca mprove the model. The predcted vs. actual plot of lear SVM, quadratc SVM, ad coarse Gaussa SVM are show Fg. 6, Fg. 7, ad Fg. 8, respectvely. Fg. 7 The Predcted vs. Actual plot of quadratc SVM. Fg. 8 The Predcted vs. Actual plot of coarse Gaussa SVM. 4.3 Evaluate Model Usg Resduals Plot We further evaluate the model performace by usg the resduals plot after trag a model. The resduals plot dsplays the dfferece betwee the predcted ad true resposes. Usually a good model has resduals scattered roughly symmetrcally aroud 0. If we ca see ay clear patters the resduals, t s lkely that we ca mprove the model. We eapecally look for the followg patters: Resduals are ot symmetrcally dstrbuted aroud 0. Resduals chage sgfcatly sze from left to rght the plot. Outlers occur, that s, resduals that are much larger tha the rest of the resduals. Fg. 6 The Predcted vs. Actual plot of lear SVM. E-ISSN: 4-3488 89 Volume 13, 017
Na Zhag, Keea Leatham Clear, olear patter appears the resduals. The resdual plots of lear SVM, quadratc SVM, ad coarse Gaussa SVM are show Fg. 9, Fg. 10, ad Fg. 11, respectvely. Fg. 9 The resduals plot of lear SVM. 4.4 Model Statstcs The model parameters are very useful ad mportat to evaluate the performace of dfferet models. They are defed as follows. RMSE (Root mea square error). The RMSE s always postve ad ts uts match the uts of the respose. Look for smaller values of the RMSE. R-Squared. Coeffcet of determato. R- squared s always smaller tha 1 ad usually larger tha 0. It compares the traed model wth the model where the respose s costat ad equals the mea of the trag respose. If the model s worse tha ths costat model, the R-Squared s egatve. Look for a R- Squared close to 1. MSE (Mea squared error). The MSE s the square of the RMSE. Look for smaller values of the MSE. MAE (Mea absolute error). The MAE s always postve ad smlar to the RMSE, but less sestve to outlers. Look for smaller values of the MAE. For each SVM algorthm, after the etwork has bee well traed, we evaluate the performace of each featured subset. The comprehesve comparso s show Table 1. TABLE 1 COMPASIRON OF DIFFERENT SVM MODELS Fg. 10 The resduals plot of quadratc SVM. RSME R- Sq MSE MAE Tra Tme (sec) Lear 6.61*10-7 0.98 6.61*10-7 6.61*10-7 0.4687 Quadratc 1.79*10-6 0.86 3.3*10-1 1.46*10-6 0.118 Cubc 3.65*10-6 0.4 1.3*10-11 3.10*10-6 0.1149 Fe 3.77*10-6 0.37 1.43*10-11 3.0*10-6 0.117 Gaussa Meda 6.38*10-6 0.98 4.07*10-13 4.71*10-7 0.1059 Gaussa Coarse Gaussa 7.83*10-7 0.97 6.13*10-13 6.05*10-7 0.1444 Fg. 11 The resduals plot of coarse Gaussa SVM. E-ISSN: 4-3488 90 Volume 13, 017
Na Zhag, Keea Leatham 5 Coclusos We propose a kerel based SVM algorthm wth varable models to adapt to the hgh-dmesoal but relatvely small samples for remote explosve detecto o photo-thermal frared magg spectroscopy (PT-IRIS) classfcato. For each SVM algorthm t reveals classfcato accuracy ad mmum feature umber objectves. After the etwork has bee well traed, we evaluate the performace of each featured subset. The respose plot,predcted vs. actual plot, ad resduals plot of the lear, quadratc, ad coarse Gaussa SVM are demostrated. A comprehesve comparso of lear SVM, quadratc SVM, cubc SVM, fe Gaussa SVM, meda Gaussa SVM, coarse Gaussa SVM s performed terms of root mea square error, R-squared, mea squared error, ad mea absolute error. The excellet expermetal results demostrated that the kerel based SVM models provde a promsg soluto to hgh-dmesoal data sets wth lmted trag samples. ACKNOWLEDGMENT Ths work was supported by the Natoal Scece Foudato (NSF) grats: HRD #1505509, DUE #1654474, ad HRD #1533479. Refereces: [1] L. Zhag, Q. Zhag, B. Du, X. Huag, Y. Y. Tag ad D. Tao, "Smultaeous Spectral- Spatal Feature Selecto ad Extracto for Hyperspectral Images," IEEE Trasactos o Cyberetcs, vol. 48, o. 1, pp. 16-8, Ja. 018. [] M. Lu, C. Xu, Y. Luo, C. Xu, Y. We ad D. Tao, "Cost-Sestve Feature Selecto by Optmzg F-Measures," IEEE Trasactos o Image Processg, vol. 7, o. 3, pp. 133-1335, March 018. [3] X. We, L. Shao, W. Fag ad Y. Xue, "Effcet Feature Selecto ad Classfcato for Vehcle Detecto," IEEE Trasactos o Crcuts ad Systems for Vdeo Techology, vol. 5, o. 3, pp. 508-517, March 015. [4] C. Cao, R. L. Tutwler ad S. Slobouov, "Automatc Classfcato of Athletes Wth Resdual Fuctoal Defcts Followg Cocusso by Meas of EEG Sgal Usg Support Vector Mache," IEEE Trasactos o Neural Systems ad Rehabltato Egeerg, vol. 16, o. 4, pp. 37-335, Aug. 008. [5] T. Zhag, J. Lu, S. Lu, C. Xu ad H. Lu, "Boosted Exemplar Learg for Acto Recogto ad Aotato," IEEE Trasactos o Crcuts ad Systems for Vdeo Techology, vol. 1, o. 7, pp. 853-866, July 011. [6] Vapk, V. The Nature of Statstcal Learg Theory. Sprger, New York, 1995. [7] Fursteberg, R., Kedzora, C. A., Stepowsk, J., Stepowsk, S. V., Rake, M., Papatoaks, M. R., Nguye. V., Hubler, G. K., ad McGll, R. A., Stad-Off Detecto of Trace Explosves va Resoat Ifrared Photothermal Imagg, Appl. Phys. Lett., vol. 93, o., 008. [9] Fursteberg, R., Kedzora, C. A., Stepowsk, J., Stepowsk, S. V., Rake, M., Papatoaks, M. R., Nguye. V., Hubler, G. K., ad McGll, R. A., Stad-Off Detecto of Trace Explosves va Resoat Ifrared E-ISSN: 4-3488 91 Volume 13, 017
Na Zhag, Keea Leatham Photothermal Imagg, Appl. Phys. Lett., vol. 93, o., 008. [10] C. A. Kedzora, R. Fursteberg, M. Papatoaks, Ifrared Photothermal Imagg of Trace Explosves o Relevat Substrates, Proceedgs of SPIE, vol. 8709, 013. E-ISSN: 4-3488 9 Volume 13, 017