Terahertz Spectroscopic Material Identification Using Approximate Entropy and Deep Neural Network

Transcription

1 Terahertz Spectroscopic Materia Identification Using Approximate Entropy and Deep Neura Network Yichao Li 1, Xiaoping A. Shen 2, Robert L. Ewing 3, and Jia Li 4 1 Schoo of Eectrica Engineering and Computer Science, Ohio University Athens, OH USA 2 Department of Mathematics, Ohio University, Athens, OH 45701, USA 3 Sensors Directorate, Wright-Patterson Air Force Base, Dayton, Ohio , USA 4 Schoo of Engineering and Computer Science, Oakand University, Rochester, MI Emais: Xiaoping Shen<shenx@ohio.edu>, Jia Li< i4@oakand.edu> Abstract Terahertz spectroscopy and imaging are a rapidy deveoped technique with important appications in many areas, such as medica imaging, security, chemistry, biochemistry, astronomy, communications, and manufacturing, to name a few. However, terahertz spectroscopy and imaging produce excessivey high dimensiona data which is prohibitive for common methods deveoped in the area of image processing. In this paper, we report our recent study on a nove cassifier based on feature extraction using approximate entropy (ApEn). The cassifier is initiated by anayzing the compexity of the terahertz spectrum, which is then combined with a deep neura network for materia cassification. Experimenta resuts show that approximate entropy based features have high sensitive for detecting meta matrix composites, the accuracy of identification is up to 96.3%. Reated agorithms for ApEn feature extraction and materia cassification are iustrated. An optima parameterembedding dimension, subect to cassification accuracy for ApEn is studied. Keywords Terahertz Spectroscopy; Approximate Entropy; Deep Neura Network; Materia Cassification; Meta Identification. I. INTRODUCTION Traditionay the three maor casses of materias are metas, poymers, and ceramics. Exampes of these are stee, coth, and pottery. These casses usuay have quite different sources, characteristics, and appications. Materias are usuay cassified by their performance, physica/chemica properties, composition/structure and processing/synthesis. Nowadays, terahertz (Thz) spectroscopy and imaging have emerged as an innovative technoogy for materia cassification with unprecedented sensitivity [1]. Thz (1 Thz = 1012 Hz) radiation covers a part of eectromagnetic (EM) spectrum where waveengths range from 3 mm. to 30 m. or frequency bands range from 0.1 to 10 Thz [2]. Thz waves can penetrate most of dieectric materias, e.g. poymers, paper, wood, and organic fims [3], [4]. In the contrast, Thz radiation shows strong absorption by water, which makes it idea for cassifying norma tissues and cancer tissues since their water content are different [5]. Previous materia cassification researches based on Thz spectroscopy were studied in a sma scae in terms of number of kinds of materias. In [4], Zhong et a. used minimum distance cassifier and neura network methods to cassify 4 exposive and biochemica materias with an average accuracy above 80%. In [6], Fitzgerad et a. used principa components anaysis (PCA) and support vector machine (SVM) to cassify norma breast tissue and breast cancer tissue with an accuracy of 92%. In [7], Zou et a. showed that PCA is a feasibe method to cassify materias (water, intraipid, and gucose were used; however, cassification accuracy was not reported). This paper presents a new methodoogy for materia cassification based on Thz spectroscopy, we empoy Approximate Entropy (ApEn; it wi be introduced in next section) as a dimensionaity reduction and non-inear feature extraction method for Thz spectrum data. We then use the earnt features as inputs to a non-inear earning machine deep neura network in the hope of achieving state of the art accuracy for materia cassification probems. In this preiminary study, we have anayzed 107 sampes from 14 kinds of materias incuding 6 meta-containing materias (e.g. oxides) and 8 non-meta materias. We report that ApEn provides powerfu features for detecting metacontaining materias with an accuracy of 96.3%. We aso show, empiricay, that an optima window ength (embedding dimension) for ApEn is 4, which can generay capture key properties of Thz spectrum given an overa 14 materias cassification accuracy of 80.4%. The paper is organized as the foows, Section II provides simpe mathematica background of ApEn and deep neura network. Section III describes the experiment design and a brief introduction of the data sets used in these experiments. After that, the resuts of materia cassification are presented in Section IV. We concude the paper and briefy discuss future works in Section V. II. BACKGROUND A. Approximate Entropy (ApEn) The approximate entropy ApEn(, r) was introduced by Pincus [8], which is a statistica measure quantifying the change of compexity and reguarity for time-series data. For an N- dimensiona time series, its ApEn depends on two parameters: the windows ength (embedding dimension ), and the fitering /17/$ IEEE 52

2 eve threshod (r). A time series containing many repetitive patterns has a reativey smaer ApEn; a ess predictabe (i.e., more compex) process has a higher ApEn. ApEn can be summarized as foows (for detaied discussion, we refer readers to [8] [11]). Step 1: Form a time series of datau u u, where k 1, 2, k, is the ength of time series. Step 2: Assume integer represent the ength of compared run of data, and positive rea number r repesent a fitering eve. Step 3: Form a sequence of vectors v v in v 1, 2, ( k1) R, which is rea dimensiona sapce defined by v u, u,, u i i ( i1) u ( i1) Step 4: Use the sequence v 1, v2, v( k1) to construct, for each i, 1 i k 1 number of v such that d[ v v ] r h i h C ( r), i k 1 where d v i v ] is defined as [ h d [ vi vh] max u( iq) u( hq) q0,, 1 The signa d represents the distance between the vectors v i, vh,given by the maximum difference in their the maximum difference in their respective scaar components. Step 5: Define k 1 1 i1 ( r ) ( k 1) og( C ( r)). Step 6: Define approximate entropy as ApEn ( r) 1 ( r), Agorithms used to cacuate the approximate entropy are deveoped in [8] [11]. It can be formatted as Agorithm 1 shown beow. The agorithm takes 3 inputs: a finite series U, a window ength, and a fitering (threshoding) eve r. The output is a rea number which can be used as measurement of compexity and randomness of the input series. i. B. Deep Neura Network (DNN) Deep earning architectures have successfu appications in both visua and speech recognitions[12]. Deep earning methods aow a machine to automaticay discover intrinsic reationships needed for cassification. Deep neura networks are one of the basic types of deep earning architectures, which consists of an input ayer, 2 or more hidden ayers, and an output ayer. Each ayer consists of a ist of neurons. Learning process is through backpropagation iterations. In each backpropagation iteration, gradients of the obective function with respect to each neuron can be cacuated using the chain rue for derivatives. Therefore stochastic gradient descent (SGD) are used by most practitioners[12]. In this preiminary study, we used Adam[13], instead of SGD, to do obective optimization. Adam[13] requires itte fine tuning and it is computationay more efficient than SGD. Many we-known deep earning frameworks are avaiabe in Python, such as Theano [14], Torch, Caffe [15], TensorFow [16]. In this study, we used Keras [17], which is a high-eve deep earning API (appication programming interface) on top of Theano or TensorFow. The best feature about Keras is that the earning architecture can be impemented within a minute. The agorithm used in this study is shown beow as Agorithm 2. The eave-one-out cross vaidation is impemented using scikit-earn [18]. III. EXPERIMENT DESIGN A. Terahertz spectroscopy dataset We have coected 107 Thz spectrum sampes of 14 materias (6 meta matrix composites: Zinc, Iron, Cobat, Cadmium, Mercury, and Copper; 8 non-meta materias: Tetracene, Poyfim, Poyethyene, Terephthaic Acid, Asphat, Trehaose, Mik powder, Coffee) from the Thz database ( The dimensionaity of Thz spectrums ranges from 795 to The Thz database is currenty the most comprehensive Thz spectroscopic database estabished by the NICT and RIKEN in Japan[19]. 53

3 B. Materia cassification Most of the sampes have a arge range of eectromagnetic spectrum, from 0 19 Thz (10 12 Hz). To identify optima window size (denoted as L opt) for materia cassification, we study the impacts of the parameter L with 36 different choices of r. Specificay, for each of the window size, L=2, =0, 1, 2, 3, there are 36 different fitering eves k*10 i-4, i=0, 1, 2, 3, k=1,,9. For each of the 4 settings, we appy the same deep neura network to do cassification. We use eave one out cross vaidation to evauate the cassification accuracy. An overa workfow is shown in Figure 1. Figure 2: ApEn parameter seection scheme is shown. Four different vaues of window ength are used. The same deep neura network is used. Accuracy is evauated based on eave-one-out cross vaidation. Optima window ength is subect to maxima accuracy. Figure 1: A deep neura network scheme is shown. The terahertz spectrum is first anayzed using approximate entropy, which is a feature extraction method to capture key properties (e.g. compexity, periodicity) of the spectrum. Totay 36 features are obtained by fixing the window ength (L=2, =0,1,2,3) and varying the fitering eves to k*10 i-4, i=0,1,2,3, k=1,,9. Two hidden ayers are used, each with 100 neurons and 0.1 dropout rate. The output ayer contains 14 neurons for each cass. C. Parameter Seection for ApEn There are two parameters for ApEn. One is the window ength, which is reated to the embedding dimension of a time series. An overapping partition of a time series with interva size is determined. The compexity (use entropy as a measure) on each subinterva is cacuated and compared to the consecutive subinterva to detect the change of compexity. The other one is the fitering eve r, which contros the threshod of statistica bias. In this study, we focused on identifying the optima window ength, denoted as, L opt. A simpified parameter seection fowchart is shown in Figure 2. Experiment resuts and discussion for threshod parameter wi be reported in near future. D. Equipment A singe PC was used for both approximate entropy feature extraction and deep neura network cassification. The operating system is Windows 7 Service Pack 1. The CPU is Inte Core 3.50 GHz * 8. The memory is 16GB. Deep neura network cassification was performed using Graphics Processing Unit (GPU) in GeForce GTX 770. The whoe eave-one-out cross vaidation process can be done within 2 hours. IV. RESULTS Experimenta resuts show that approximate entropy based features have high sensitive for detecting materias that contain metas, the accuracy of identification is up to 96.3%. Reated agorithms for the cassifier are iustrated above. Other experimenta resuts reated to threshoding parameter r, agorithmic deveopment and computation compexity anaysis wi be reported in future. A. Identification of optima window ength for ApEn Our resuts show that the optima window ength is 4 for Thz spectrums (Figure 3). The overa best accuracy for 14 materia cassification is 80.4%. Figure 3: Cassification accuracy when different ApEn window ength is used. The accuracy is 77.6%, 78.5%, 80.4%, 78.5% for window ength of 1, 2, 4, 8, respectivey. 54

4 techniques, in particuar, these waves penetrate amost a nonmetaic or poarizing substances [19]. Thz waves aso stimuate moecuar and eectronic motions in many materias - refecting off some, propagating through others, and being absorbed by the rest. Benefit to these properties, Thz time - Tabe 1: Confusion matrix for cassification of 14 materias using approximate entropy with optima window ength of 4 and fitering eve ranging from k*10 i-4, i=0,1,2,3, k=1,,9. Rows are actua casses. Coumns are predicted casses. Grey fied ces are materias containing meta. Doube ine ces are nonmeta materias. Diagonas are the number of correcty identified materias (boded). Miscassified materias are marked using asterisk. The accuracy for detecting metas is 96.3%. The overa materia cassification accuracy is 80.4%. B. Approximate Entropy provides high accuracy for detecting meta-containing materias To iustrate the miscassified materias, a cassification confusion matrix based on optima window ength is shown in Tabe 1. Fourteen materias are shown; the rows are actua casses and the coumns are predicted casses. Diagona ces are boded, indicating that they are correcty identified materias. Miscassified materias are abeed using asterisk. In terms of cassification of metas vs. non-metas, our resuts show that ApEn features are very powerfu, with an accuracy of 96.3%. Indeed, ApEn can capture the reguarity and compexity of the spectrum, and metas and nonmetas do have arge differences in their physica and chemica properties. Notaby, no sampes from meta group (mercury, iron, or copper) are correcty separated from other members in the same group. In fact, most of them are predicted as another meta composite; whie ony 3 out of 5 mercury materias are cassified as poyfim and trehaose. Intriguingy, one terephthaic acid sampe is cassified as cadmium. These observation suggest that there is a great potentia improvement for cassifying among materias within the meta group. C. Cassification accuracy varies with respect to number of casses Since muti-cass cassification probem is generay a much harder probem than binary cassification probem, we ask how the accuracy wi change with respect to the number of casses (by random grouping of existing casses). Figure 4 shows that, as expected, the accuracy drops when the number of casses increases. Based on this observation, we are working on deveoping a mutiscae cassifier with higher sensitivity for identify mutipe casses. In order to benefit from the Thz images technoogy, the new mutiscae identifier wi be constructed to refect information at moecuar eve. V. CONCLUSION Terahertz spectroscopy has found appications in many areas, such as materia science, bioogy and medicine to name a few. Thz spectroscopy has advantages over other imaginary Figure 4: Cassification accuracy varies with respect to the number of casses. Error bars are shown for each point. When number of casses increases, the accuracy wi genera go down. -domains by providing a measurement of intensity for the eectric fied produced by Thz waves, which can be used to identify sampe structure [20]. This paper reports our initia study on the materia cassification using the Thz spectroscopy using non-inear earning agorithm DNN. The novety ays in deveoping a supervised feature seection with vaidation framework based on ApEn for DNN cassifier. ApEn has two unknown parameters,, window ength, which is infuenced by the ength of the sampe, and r, threshoding parameter, which depends on randomness and quaity of data. In most of practica probems, the recommended vaues of r, in the range of times the standard deviation of the signa, have been shown to be appicabe for a wide variety of signas. However, in certain cases, r vaues within this prescribed range can ead to an incorrect assessment of the compexity of a given signa [21]. Our future work wi focus on data preprocessing and deveoping a section rue for parameter r taiored for the materia cassification probems using Thz spectroscopy imaging to enhance the accuracy of the cassifier. ACKNOWLEDGMENT The second author is sponsored by Air Force Office of Scientific Research (AFOSR) summer facuty feowship program. 55

5 REFERENCES [1] C. Baas, Review of biomedica optica imaging a powerfu, noninvasive, non-ionizing technoogy for improving in vivo diagnosis, Meas. Sci. Techno., vo. 20, no. 10, p , [2] D. Abbott and X. C. Zhang, Specia Issue on T-Ray Imaging, Sensing, and Retection, Proc. IEEE, vo. 95, no. 8, pp , Aug [3] N. FUSE, T. FUKUCHI, M. MIZUNO, and K. FUKUNAGA, High- Speed Underfim Corrosion Imaging Using a Terahertz Camera, Eectron. Commun. Japan, vo. 99, no. 8, pp , [4] H. Zhong, A. Redo-Sanchez, and X.-C. Zhang, Identification and cassification of chemicas using terahertz refective spectroscopic focapane imaging system, Opt. Express, vo. 14, no. 20, pp , Oct [5] X. Yang et a., Biomedica Appications of Terahertz Spectroscopy and Imaging, Trends Biotechno., vo. 34, no. 10, pp , [6] A. J. Fitzgerad, S. Pinder, A. D. Purushotham, P. O Key, P. C. Ashworth, and V. P. Waace, Cassification of terahertz-pused imaging data from excised breast tissue, J. Biomed. Opt., vo. 17, no. 1, pp , [7] Y. Zou, P. Sun, and W. Liu, Cassification of materias using terahertz spectroscopy with principa components anaysis, in th Internationa Conference on Infrared, Miimeter, and Terahertz waves (IRMMW-Thz), 2015, pp [8] S. M. Pincus, Approximate entropy as a measure of system compexity., Proc. Nat. Acad. Sci., vo. 88, no. 6, pp , [9] S. M. Pincus and A. L. Godberger, Physioogica time-series anaysis: what does reguarity quantify?, Am. J. Physio., vo. 266, no. 4 Pt 2, pp. H , Apr [10] S. M. Ryan, A. L. Godberger, S. M. Pincus, J. Mietus, and L. A. Lipsitz, Gender- and age-reated differences in heart rate dynamics: are women more compex than men?, J. Am. Co. Cardio., vo. 24, no. 7, pp , Dec [11] K. K. Ho et a., Predicting surviva in heart faiure case and contro subects by use of fuy automated methods for deriving noninear and conventiona indices of heart rate dynamics, Circuation, vo. 96, no. 3, pp , Aug [12] Y. LeCun, Y. Bengio, and G. Hinton, Deep earning, Nature, vo. 521, no. 7553, pp , May [13] D. P. Kingma and J. Ba, Adam: {A} Method for Stochastic Optimization, CoRR, vo. abs/ , [14] R. A-Rfou et a., Theano: A {Python} framework for fast computation of mathematica expressions, arxiv e-prints, vo. abs/ , May [15] Y. Jia et a., Caffe: Convoutiona Architecture for Fast Feature Embedding, arxiv Prepr. arxiv , [16] M. Abadi et a., {TensorFow}: Large-Scae Machine Learning on Heterogeneous Systems [17] F. Choet and others, Keras. GitHub, [18] F. Pedregosa et a., Scikit-earn: Machine Learning in {P}ython, J. Mach. Learn. Res., vo. 12, pp , [19] T. Notake, R. Endo, K. Fukunaga, I. Hosako, C. Otani, and H. Minamide, State-of-the-Art Database of Terahertz Spectroscopy Based on Modern Web Technoogy, IEEE Trans. Terahertz Sci. Techno., vo. 4, no. 1, pp , Jan [20] S. Wang and X.-C. Zhang, Pused terahertz tomography, J. Phys. D. App. Phys., vo. 37, no. 4, p. R1, [21] S. Lu, X. Chen, J. K. Kanters, I. C. Soomon, and K. H. Chon, Automatic Seection of the Threshod Vaue R for Approximate Entropy, IEEE Trans. Biomed. Eng., vo. 55, no. 8, pp , Aug