Estimation of Human Emotions Using Thermal Facial Information

Transcription

1 Estimation o Human Emotions Using Thermal acial Inormation Hung Nguyen 1, Kazunori Kotani 1, an Chen 1, Bac Le 2 1 Japan Advanced Institute o Science and Technology, 1-1 Asahidai, Nomi, Ishikawa, Japan 2 University o Science, 227 Nguyen Van Cu, Ho Chi inh city, Vietnam ABSTRACT In recent years, research on human emotion estimation using thermal inrared (IR) imagery has appealed to many researchers due to its invariance to visible illumination changes. Although inrared imagery is superior to visible imagery in its invariance to illumination changes and appearance dierences, it has diiculties in handling transparent glasses in the thermal inrared spectrum. As a result, when using inrared imagery or the analysis o human acial inormation, the regions o eyeglasses are dark and eyes thermal inormation is not given. We propose a temperature space method to correct eyeglasses eect using the thermal acial inormation in the neighboring acial regions, and then use Principal Component Analysis (PCA), Eigen-space ethod based on class-eatures (EC), and PCA-EC method to classiy human emotions rom the corrected thermal images. We collected the Kotani Thermal acial Emotion (KTE) database and perormed the experiments, which show the improved accuracy rate in estimating human emotions. Keywords: Thermal acial inormation, estimation human emotion, principal component analysis, eigen-space method based on class-eature, KTE database. 1. INTRODUCTION Nowadays, Human Computer Interaction (HCI) is a very attractive research area in computer vision. One o the key researches in HCI is to detect the inner emotions through human aces by perorming automatic analysis o acial expressions. any previous works have proposed towards developing acial emotion estimation [1]. However, we still lack an accurate and robust acial emotion estimation method to be deployed in uncontrolled environments. Several actors that aect acial emotion estimation include pose variations, occlusions, and most importantly, illumination changes [1]. Thereore, it is a new and imaginative way to use IR imagery, which is not sensitive to light condition, to ill the gap in the human emotion estimation ield. Besides, human emotions could be maniested by changing temperature o ace skin which is obtained by IR camera. Consequently, thermal inrared imagery gives us more inormation to help us robustly estimate the human emotions. Recently, a number o studies have demonstrated that thermal inrared imagery oers a promising alternative to visible imagery in acial emotion estimation problems by better handling the visible illumination changes. Y.Yoshitomi et al. used two dimensional detection o temperature distribution on the ace using inrared rays [2]. Based on studies in the ield o psychology, several blocks on the ace are chosen or measuring the local temperature dierence. With Back Propagation Neutral Network, the acial expression is recognized. The recognition accuracy reaches 90% with neutral, happy, surprising and sad expressions. However, the testing database is obtained rom only one emale rontal view. Y. Yoshimomi generated eature vectors by using a two-dimensional Discrete Cosine Transormation (2D-DCT) to transorm the grayscale values o each block in the acial area o an image into their requency components, and used them to recognize ive expressions, including angry, happy, neutral, sad, and surprise. The mean expression accuracy is 80% with our test subjects [3]. Y.Koda et al. used the idea rom [3] and added a proposed method or eiciently updating o training data, by only updating the training data with happy and neutral acial expression ater an interval [4]. The expression accuracy increased rom 80% to 87% with this new approach. Sophie Jarlier et al. extracted the eatures as representative temperature maps o nice action unit (AUs) and used K-nearest neighbor to classiy seven expressions [5]. The database or testing has our persons and the accuracy rate is 56.4%...Khan et al. suggested using acial Thermal eature Points (TPs), which are deined as acial points that undergo signiicant thermal changes in presenting an expression, and used Linear Discriminant Analysis (LDA) to classiy intentional acial expressions based on Thermal Intensity Values (TIVs) recorded at the acial Thermal eature Points (TPs) [6]. The database has 16 persons with 5 expressions and the accuracy rate ranges rom 66.3% to 83.8%. L.Trujillo et al. proposed using a local and global automatic eature localization procedure to perorm acial expression in thermal images. They ith International Conerence on Graphic and Image Processing (ICGIP 2013), edited by Yulin Wang, Xudong Jiang, ing Yang, David Zhang, Xie Yi, Proc. o SPIE Vol. 9069, 90690O 2014 SPIE CCC code: X/14/$18 doi: / Proc. o SPIE Vol O-1

2 used PCA to reduce the dimension and interest point clustering to estimate acial eature localization and Support Vector achine (SV) to classiy three expressions [7]. B.Hernandez et al. used SV to classiy the expressions surprise, happy, neutral rom two inputs. irst input consists o selection o a set o suitable regions where the eature extraction is perormed, second input is the Gray Level Co-occurrence atrix used to compute region descriptors o the IR images [8]. B.R.Nhan et al. extracted time, requency and time-requency eatures rom thermal inrared data to classiy the natural responses in terms o subject-indicated levels o arousal and valence stimulated by the International Aective Picture System [9]. All these studies with thermal inrared imagery have shown that human emotion states are related to the acial skin temperature property. However, to our knowledge, most approaches that use the extracted eatures rom a single inrared thermal image may lose some useul inormation which could be contained in the sequences. Although there are many signiicant advantages when we use thermal imagery, IR have several drawbacks. One o drawbacks is eyeglasses. Glass is opaque to IR and the sensitivity o thermal IR to acial occlusions is decreased by eyeglasses. This is because objects made o glasses act as a temperature screen, completely occluding the parts located behind them. To eliminate the eect o presence o eyeglasses, we propose a temperature space method or thermal inrared data. We will also reduce the impact o ambient temperature by normalizing it between rames. To estimate ive emotions, we use PCA, PCA-EC, and EC to extract eature vectors and then ind the similarity between the testing data and training data. 2. ETHOD In this section, we propose a method to reduce the eect o eyeglasses in thermal data and then use three conventional methods PCA, EC, and PCA-EC to analyze emotions. Beore applying temperature spaces, to avoid the temperature change o ambience rom rame to rame, we calculate the mean o ambient point temperatures o each rame. Then we ind the dierence o the mean o each rame to that o its previous rame and update the temperature o all points o each rame by subtracting those temperatures with the dierence. 2.1 Temperature spaces We propose using temperature spaces to support the temperature analysis o human ace area. Based on temperature data and observation, we classiy images into two main spaces, i.e. non-ace space and ace space. Non-ace space is or the background area, which urther includes two sub-spaces. In ace space, there are three sub-spaces because the eyebrows and nose are usually cold whereas the cheek is warm [10], and the orehead is usually the warmest. Using the spaces, we ind eyeglasses area and replace each point o them by the mean o image temperature. Let h and g be maps rom image to temperature space and image to glass space respectively, as the ollowing: h : I T g : I G g h i, j hi, j i, j g hi, j i, j where I is image space, T is temperature space, and G is glass space. We calculate the temperature spaces or image and ace as ollowing: 0 i i, j,, glass h i j i i j glass I I T T T ; T T / 5 I ax in I I I I where T, T are maximum and minimum o temperature o each image, respectively. ax in, / * 1, * 5,, L i j T T k i j T T k i j I I I I k in I ax I where k 1,5 T T T ; T T / 5 ax in where T, T are maximum and minimum o temperature o each human ace area, respectively. ax in, / * 1, * 5,, where i 1,5 i in ax L i j T T i i j T T i i j Proc. o SPIE Vol O-2

3 Based on (3), (5), (6), we ind the glass area as ollowings: L0 i, j / i, j Tin, i, j (6) glass i, j / i, j max min L I 2,min L 0,min max L I 3,max L 1 (7) Ater using our proposed method to ind eyeglasses area in thermal data, to reduce the eect o eyeglasses, we replace the temperature o the areas by the mean o image temperature. 2.2 Estimate emotion using PCA, EC, and PCA-EC Ater reducing the eect o eyeglasses, we use three conventional methods PCA, EC, and PCA-EC to estimate the human emotion using thermal inormation o emotion With PCA, the aim is to build a ace space, including the basis vectors called principal components, which better describes the ace images [11]. To estimate emotions using PCA, we divide the training set into ive classes and compute the eigen-space o each class as ollowing: Step 1: Each thermal data rame is shaped rom 2D matrix to 1D vector. Given rames o thermal data as training data, we convert these datum to corresponding column vectors. Step 2: ind covariance matrix that represents the scatter over the mean o training data 1 T 1 S x x x x ; x x k k k k1 k1 (8) where x k is an N-dimension vector, is total number o rames o each class and S is covariance matrix. Step 3: Solve the eigenvalues problem S= and obtain the eigenvalues and corresponding eigenvectors. Step 4: Choose the largest H eigenvalues and eigenvectors. Eigen-space includes H eigenvectors. or each test thermal data, we project it to the eigen-space o each class and derive the reconstruction thermal data rom each eigen-space. Using mean square error, measuring the similarity, between input thermal data and reconstruction thermal data, we can choose a suitable class or input thermal data which is a minimum o the mean square errors [11]. In [12] and [13], the authors suggest using Eigen-space ethod based on Class-eatures (EC) to analyze the acial expressions. The dierence between PCA and EC is that PCA inds the eigenvector to maximize the total variance o the projection to line, while EC obtains eigenvector to maximize the dierence between the within-class and betweenclass variance. The dierence between the within-class and between-class variance is calculated as ollowing: T 1 1 S S S ; S x x x x ; S x x x x T B W B W (9) m m m1 1 1 x x ; x x m m m1 m1 (10) where is a set o expression classes, acial-patterns are given or each class and x m is an N-dimension vector o the m-th acial pattern, m 1,. To estimate emotion using EC, we divide the training set into ive classes and compute the eigen-space including eigen-vectors o dierence matrix. or each test thermal data, we project it into the eigen-space o each class and an emotion is chosen i it gets maximum o cosine o angle between obtained vector ater projection and eigenvector o each class. With PCA-EC, we use PCA to reduce the dimension and apply EC to the obtained data. 3. EXPERIENTS or database, we use KTE (Kotani Thermal acial Emotion) database [14]. This database contains 26 subjects who are Vietnamese, Japanese, Thai rom 11 year-olds to 32 year-olds with seven emotions. The example o thermal images Proc. o SPIE Vol O-3

4 is shown in ig. 1. In our experiments, we use only sequence thermal data to estimate human emotions. rom 130 GB thermal data, we extracted 2.4 GB o sequence thermal data or ive emotions. ri igure 1. Examples thermal images o ive expressions (neutral, ear, anger, happiness, sadness) [14] In our experiments, we use PCA, EC, and PCA-EC with beore and ater removal o eyeglasses. rom the obtained database, we separate the training and testing data as 60% and 40 % o the total thermal data. Table 1 shows the results with EC. The results show that accuracy increases with happy, ear, and sad emotions but remains constant with angry and neutral emotions. Table 2 shows the results with PCA. The accuracy o angry emotion increases to a large extent. However with neutral emotion, the accuracy decreases. In total, the mean accuracy rate increases. Table 3 shows the results with PCA-EC. Although the mean accuracy rate increases, the accuracy o angry and happy emotion decreases. In totality, the mean accuracy rate is only slightly increased because some inormation is lost when we reduce the dimension by using PCA. TABLE I. EC WITH BEORE AND ATER REOVAL O EYEGLASSES Ag Ha e Ne Sa Av Ag Ha e Ne Sa Av TABLE II. PCA WITH BEORE AND ATER REOVAL O EYEGLASSES Ag Ha e Ne Sa Av Ag Ha e Ne Sa Av TABLE III. PCA-EC WITH BEORE AND ATER REOVAL O EYEGLASSES Ag Ha e Ne Sa Av Ag Ha e Ne Sa Av CONCLUSION This paper describes a novel way to reduce the eect o changing ambient temperature which occurs during the data acquisition and the proposed method to reduce the eect o eyeglasses using temperature space in thermal data. To Proc. o SPIE Vol O-4

5 estimate human emotion and prove the eiciency o the proposed method, three convention methods, namely PCA, EC, PCA-EC, are used with KTE database. Since the eyeglasses areas are replaced with the averaged-ambient temperature, these is no more eect o ambient temperature to these areas. The experiment with ater removing glasses and beore removing glasses shows the increasing o accuracy rate. Specially, with PCA, the accuracy o angry emotion increases rom 59.17% to 93.33%. In general, the accuracy o each emotion and each method has been improved with the correction o eyeglass areas. However, in some case, the loss o inormation causes the decrease o accuracy rate. In the uture, to improve the eiciency o the proposed method, we intend to combine visible sequence image and thermal sequence image to have used-model. REERENCES [1] Z.Zeng,.Pantic, G.I.Roisman,T.S. Huang, A survey o aect recognition methods: Audio, visual, and spontaneous expressions, IEEE Transactions on Pattern Analysis and achine Intelligence 31, vol 1, pp , (2009). [2] Y.Yoshitomi, N. iyawaki, S.Tomita, S. Kimura, acial expression recognition using thermal image processing and neural network, Robot and Human Communication, RO-AN '97 Proceedings, 6th IEEE International Workshop, pp , (1997). [3] Y.Yoshitomi, acial expression recognition or speaker using thermal image processing and speech recognition system, Proceedings o the 10th WSEAS International Conerence on Applied Computer Science, pp , (2010). [4] Y.Koda, Y.Yoshitomi,. Nakano,.Tabuse, A acial expression recognition or a speaker o a phoneme o vowel using thermal image processing and a speech recognition system, The 18th IEEE International Symposium on Robot and Human Interactive Communication, RO-AN 2009, pp , (2009). [5] S.Jarlier, D.Grandjean, S.Delplanque, K. N Diaye, I.Cayeux,. Velazco, D.Sander, P.Vuilleumier, K.Scherer, Thermal analysis o acial muscles contractions, IEEE Transactions on Aective Computing, vol. 2, pp. 2-9, (2011). [6].Khan,R.D.Ward,.Ingleby, Classiying pretended and evoked acial expressions o positive and negative aective states using inrared measurement o skin temperature, AC Transactions on Applied Perception, vol. 6, pp. 1 22, (2009). [7] L.Trujillo, G.Olague, R.Hammoud, B.Hernandez, Automatic eature localization in thermal images or acial expression recognition, IEEE Computer Society Conerence on Computer Vision and Pattern Recognition- Workshops, CVPR Workshops, p. 14, (2005). [8] B.Hernández, G.Olague, R.Hammoud, L.Trujillo, E. Romero, Visual learning o texture descriptors or acial expression recognition in thermal imagery, Computer Vision and Image Understanding, vol. 106, pp , (2007). [9] B.R.Nhan, T.Chau, Classiying aective states using thermal inrared imaging o the human ace, IEEE Transactions on Biomedical Engineering, vol. 57, pp , (2010). [10] S.Wang, Z.Liu, P.Shen,Q.Ji, Eye localization rom thermal inrared images, Patter Recognition, (2013). [11] D.T.Lin, acial Expression Classiication Using PCA and Hierarchical Radial Basis unction Network, Journal o Inormation Science and Engineering, vol. 22, pp , (2006). [12] T.Kurozumi, Y.Shinza, Y.Kenmochi and K.Kotani, acial Individuality and Expression Analysis by Eigenspace ethod Based on Class eatures or ultiple Discriminant Analysis, Proc.o ICIP 99, vol. 1, pp , (1999). [13] T.Yabui, Y.Kenmochi, K.Kotani, acial expression analysis rom 3D range images; comparison with the analysis rom 2D images and their integration, Pro.o ICIP 2003, vol.2 pp , (2003). [14] H.Nguyen, K.Kotani,.Chen, B.Le, A thermal acial emotion database and its analysis, PSIVT 2013 (Accepted). Proc. o SPIE Vol O-5