Performance Evaluation of Acoustic Scene Classification Using DNN-GMM and Frame-Concatenated Acoustic Features

Transcription

1 Proceeing of APSIPA Annual Summit an Conference 2017 Performance Evaluation of Acoutic Scene Claification Uing NN-GMM an Frame-Concatenate Acoutic Feature Gen Takahahi, Takehi Yamaa, Nobutaka Ono an Shoji Makino Univerity of Tukuba, Japan National Intitute of Informatic / SOKENAI, Japan g.takahahi@mmlab.c.tukuba.ac.jp Abtract e previouly propoe a metho of acoutic cene claification uing a eep neural network-gauian mixture moel (NN-GMM) an frame-concatenate acoutic feature. It wa ubmitte to the etection an Claification of Acoutic Scene an Event (CASE) 2016 Challenge an wa ranke eighth among 49 algorithm. In the propoe metho, acoutic feature in temporally itant frame were concatenate to capture their temporal relationhip. The experimental reult inicate that the claification accuracy i improve by increaing the number of concatenate frame. On the other han, the frame concatenation interval, which i the interval with which the frame ue for frame concatenation are electe, i another important parameter. In our previou metho, the frame concatenation interval wa fixe to 100 m. In thi paper, we optimize the number of concatenate frame an the frame concatenation interval for the previouly propoe metho. A a reult, it wa confirme that the claification accuracy of the metho wa improve by 2.61% in comparion with the reult ubmitte to the CASE I. INTROUCTION Attempt have been mae to automatically recognize human behavior an urrouning circumtance. Thi technology i applicable to monitoring elerly people, the auto-tagging of multimeia content an life-log collection. Acoutic event etection an acoutic cene claification have been focue on a funamental technologie ue in thee ytem. Acoutic event etection etect acoutic event incluing oun ignal an their timetamp, where acoutic event are a ingle oun emitte from one oun ource, uch a oor opening, coughing an moue clicking. On the other han, acoutic cene claification claifie the place or ituation where the acoutic oun wa recore. The length of an acoutic oun i typically on the orer of 10, an type of oun inclue thoe of bue, park an crow. In thi reearch, we focu on acoutic cene claification. The etection an Claification of Acoutic Scene an Event (CASE) 2013 [1] i a workhop on acoutic event etection an acoutic cene claification. Tak involving acoutic cene claification an acoutic event etection are prepare at the CASE In the metho propoe for acoutic cene claification [2][3], mel frequency pectral coefficient (MFCC), the mel frequency pectrum an recurrence quantification analyi (RQA) have been ue a feature an a Gauian mixture moel (GMM) an a upport vector machine (SVM) have been ue a claifier. The reulting claification accuracy wa at mot 70%. On the other han, eep neural network (NN), which have a multilayer neural network, have recently been actively invetigate. In general, a NN ten to fall into a local olution an require an unrealitic learning time. However, a pre-training metho [4] that give appropriate initial value an high-pee computation on a graphic proceing unit (GPU) have been etablihe. Becaue of thi, NN are now being applie in variou claification problem. For peech recognition, a NN-hien Markov moel (HMM), which combine a NN an an HMM, ha been propoe [4]. The probability itribution in an HMM i approximately repreente by a GMM. On the other han, it i preciely repreente by the NN in a NN-HMM. It wa reporte that the performance of peech recognition i markely improve uing a NN-HMM [4]. Previouly, we propoe a metho of acoutic cene claification uing a NN-GMM an frame-concatenate acoutic feature [5]. It wa ubmitte to the CASE 2016 Challenge [6] an wa ranke eighth among 49 algorithm. In the propoe metho, feature in temporally itant frame were concatenate to capture their temporal relationhip. The experimental reult inicate that the claification accuracy i improve by increaing the number of concatenate frame. On the other han, the frame concatenation interval, which i the interval with which the frame ue for frame concatenation are electe, i another important parameter. In our metho, the frame concatenation interval wa fixe to 100 m. In thi paper, we optimize the number of concatenate frame an the frame concatenation interval of the previouly propoe metho. II. PROPOSE METHO A. Proce flow of the propoe metho Figure 1 how the proce flow of the previouly propoe metho. e aume two input channel becaue the acoutic ata in the CASE 2016 ha two channel. Firt, we compute acoutic feature (the MFCC an it firt an econ ifference) in each time frame for both the left an right channel. MFCC are a repreentation of frequency characteritic coniering human auitory characteritic. The

2 Proceeing of APSIPA Annual Summit an Conference 2017 Fig. 1. Proce flow. Fig. 2. Example of a NN-GMM. feature ue in our metho are imilar a thoe in the baeline ytem of the CASE Next, we concatenate acoutic feature in each frame an channel, which we will ecribe in Sect. II-C. Finally, we perform acoutic cene claification by inputting the high-imenional acoutic feature obtaine in thi manner into the NN-GMM. e ecribe the NN- GMM below. B. NN-GMM Figure 2 how an example of a NN-GMM. A NN- GMM conit of multilayere neural network an GMM correponing to iniviual acoutic cene. A NN-GMM i baically the ame a a NN-HMM but a GMM (a onetate HMM without a tate tranition) i ue intea of an HMM. Acoutic cene claification i the problem of claifying an acoutic cene ŝ from a time erie of feature X. ŝ i expree a ŝ = argmax P ( k X), (1) k where k i an iniviual acoutic cene. By tranforming P ( k X) in thi equation uing Baye theorem, the following equation i obtaine: P ( k X) = P (X k) P (X) P ( k). (2) Since P (X) i inepenent of k, P (X) can be regare a a contant. Alo, if we aume that the probability of appearance of each acoutic cene ha a uniform itribution, we alo can ignore P ( k ). Therefore, Equation (1) become ŝ = argmax P (X k ). (3) k A GMM i often ue in acoutic cene claification a a moel for olving Equation (3). Let x t an t k be the feature vector an the tate at time frame t, repectively, then P (X k ) can be obtaine uing a GMM a follow: P (X k ) = t P (x t t k)p ( t k t 1 k ). (4) Thi repreent the probability that the erie of feature X i generate from the GMM of the acoutic cene k. It i generally ifficult to obtain the true itribution of the output probability P (x k ); thu, the GMM repreent it by the following Gauian mixture itribution: p(x k ) = I π i N(x µ i, Σ i ), (5) i=1 where N( ) i a normal itribution, an µ i an Σ i are the mean an variance of the normal itribution, repectively. Alo, I i the number of itribution ue for mixing an π i i the weight for each itribution. A NN-GMM repreent thi output probability uing a NN intea of a Gauian mixture itribution. To expre the output probability uing a NN, P (x t t k ) in Equation (4) i tranforme by Baye theorem to obtain p(x t t k) = P (t k x t) P ( t k ) P (x t). (6) Since P (x t ) i inepenent of k, P (x t ) can be regare a a contant. Alo, if we aume that the probability of appearance of each acoutic cene ha a uniform itribution, P ( t k ) can alo be ignore. The NN an GMM are integrate by etting the input vector of the NN to x t in Equation (6) an each noe of the output layer to P ( t k x t). e ecribe the learning metho of the NN-GMM aopte in thi paper. Firt, we perform upervie learning of each acoutic cene moel uing a conventional GMM. Next, uing thi GMM, we make training ata for NN coniting of the feature in each frame of each training ata an the tate number of the GMM. In the cae of acoutic cene claification, one acoutic oun file correpon to one acoutic cene, allowing thi tep to be implifie. Then, the initial parameter of the NN are etermine by unupervie pre-training [7]. In the pre-training, each layer i regare a a retricte Boltzmann machine (RBM) an the training i performe uing the contrative ivergence (C) metho. Finally, we perform fine-tuning, which i upervie training

3 Proceeing of APSIPA Annual Summit an Conference 2017 TABLE I OVERVIE OF THE EVELOPMENT ATASET AN THE EVALUATION ATASET. Fig. 3. Example of the time frame concatenation (n = 3 an m = 100 m). # of cene 15 # of oun ata of 1170 evelopment ataet =15 cene 78 ata # of oun ata of 390 evaluation ataet (=15 cene 26 ata) ata length 30 # of channel 2 (left an right) ampling frequency 44.1 khz quantization bit 16 bit uing training ata. In the fine-tuning, we a a oftmax layer initialize uing a ranom ee an perform backpropagation uing the tochatic graient ecent (SG) metho. C. Feature concatenation In the cae of acoutic cene claification, the relationhip between oun that are more temporally itant than thoe involve in peech recognition i coniere to be involve in the claification. Therefore, improvement in the claification accuracy can be expecte by concatenating uch feature. Furthermore, we concatenate the feature of the left an right channel in each time frame to capture the patial information. e ue the above feature a the input into the NN-GMM. Here we ecribe the concrete proce of frame concatenation. e concatenate acoutic feature in each frame with thoe in everal temporally itant frame before an after the frame for both the left an right channel. There are two important parameter in thi proce. One i the number of concatenate frame n an the other i the frame concatenation interval m. e concatenate n frame before an after each frame at an interval of m m, incluing the current frame. Figure 3 how an example of the time frame concatenation. hen n an m are et to 3 an 100 m, the three frame are concatenate a hown in Figure 3. On the other han, when m i change to 20 m, the eleven frame mut be concatenate. The imenion of the feature then increae about by four time. Thi woul have a negative effect to train the claifier. By uing n an m, we can capture the relationhip of temporally itant frame without increaing the imenion of the feature ignificantly. A. Experimental conition III. EVALUATION In thi experiment, we evaluate the effectivene of our metho uing the evelopment ataet an evaluation ataet provie by the CASE 2016 Challenge. Table I give an overview of thi ataet. The ataet contain 15 acoutic cene, for example, bu, retaurant an park. Each cene ha 78 oun ata in the evelopment ataet an 26 oun ata in the evaluation ataet, each of which i a tereo ignal with a uration of 30. The evaluation ataet ha open oun ata, which are ifferent from the oun ata of the evelopment ataet. The ampling frequency i 44.1 khz an the number of quantization bit i 16. TABLE II CONITIONS OF THE ACOUSTIC FEATURES AN THE NN-GMM. 20th-orer MFCC + + feature (60 imenion) n frame 2 channel frame length 40 m frame perio 20 m # of concatenate frame n 1, 3, 5, 7 frame concatenation 20, 100, 200, interval m (m) 500, 1000, 2000 # of hien layer 2, 3, 4, 5 imenion of hien layer 256, 512, 1024, 2048 imenion of input layer 120 n imenion of output layer 15 Table II how the conition of the acoutic feature an the NN-GMM. The feature are bae on a total of 60 imenion of 20th-orer MFCC an their firt an econ ifference. The time frame length an frame perio in the frame analyi are 40 an 20 m, repectively, which are the ame a thoe ue in the baeline ytem of the CASE 2016 Challenge. By the concatenation of n frame an two channel, the imenion of the feature become 20 3 n (frame) 2 (ch). The frame ue for frame concatenation are electe at m m interval. In thi experiment, the number of concatenate frame n i et to 1, 3, 5 or 7, an the frame concatenation interval m i et to 20, 100, 200, 500, 1000 or 2000 m. The number of hien layer of the NN (excluing the input layer an oftmax layer) i et to 2, 3, 4 or 5, an the imenion of each hien layer i contant an et to 256, 512, 1024 or e performe the pre-training proceing on the RBM of the hien layer uing the C-1 metho an on the RBM of the input layer uing the C-2 metho. e et the learning rate of the RBM to 0.4, the learning rate of the NN to an the ropout rate to 0.0 on the bai of the reult of a preliminary experiment. In the training of the claifier, we firt performe four fol cro-valiation (ata open, acoutic cene cloe) on the evelopment ataet an optimize the number of layer of the NN an the imenion of the hien

4 Proceeing of APSIPA Annual Summit an Conference 2017 Y Y Z E Y Y, Z & y Fig. 4. Reult of the CASE 2016 Challenge [9]. TABLE III CLASSIFICATION ACCURACY FOR EACH COMBINATION OF n AN m, ITH THE NUMBER OF HIEN LAYERS AN THE IMENSION OF THE HIEN LAYERS OPTIMIZE FOR THE EVELOPMENT ATASET IN PARENTHESES. 20 m (3, 1024) 100 m 200 m 500 m 1000 m 2000 m (2, 512) (3, 2048) (3, 512) (2, 512) (2, 256) (3, 1024) (3, 512) (3, 2048) (2, 256) layer, with a fixe ranom ee for initialization of the oftmax layer. Then, uing the parameter, we traine the claifier of which we ue all the acoutic ata in the evelopment ataet for the training, with each of ten ifferent ranom ee for initialization of the oftmax layer. e evaluate the effectivene of the metho uing the traine claifier an the evaluation ataet. e ue the Kali toolkit [8] to buil our ytem. B. Reult Table III how the reult of the experiment. The column are the time frame concatenation interval m an the row are the number of concatenate frame n. Each claification accuracy i the average of the claification accuracie obtaine for the ten ifferent ranom ee for initialization of the oftmax layer. The tanar eviation of the claification accuracie obtaine for the ten ifferent ranom ee wa at mot 1.1% in each combination of n an m. The number of hien layer an the imenion of the hien layer optimize for evelopment ataet are written in parenthee. e can ee that the claification accuracy of our metho varie with n an m. The highet average accuracy of 86.97% wa obtaine when n = 3 an m = 500 m, which i 2.38% higher than that when n = 1 (no frame concatenation). Thi emontrate the effect of uing the concatenate acoutic feature. In Table III, claification accuracie of more than 85.0%, written in TABLE IV CLASSIFICATION ACCURACY FOR EACH SOUN SCENE USING THE BASELINE SYSTEM, THE PREVIOUSLY PROPOSE METHO (n = 1) AN THE OPTIMIZE METHO (n = 3, m = 500 MS). cene metho Previouly propoe metho 1 Optimize metho 3, 500 reiential_area 83.46% 87.31% city_center 85.00% 86.15% beach 89.62% 90.38% park 91.92% 91.92% home 88.46% 86.54% foret_path 95.77% 95.00% bu 96.54% % grocery_tore 89.23% 97.31% café/retaurant 43.46% 58.85% car % % train 47.69% 57.31% tram % % metro_tation 88.46% 85.77% office % % library 69.23% 66.54% italic, were obtaine when n wa 5 or le an m wa 500 m or le. On the other han, the claification accuracy wa ignificantly reuce when n or m wa too high. Next, Figure 4 ummarize the reult of the CASE 2016 Challenge [9]. The orange, green an yellow bar in thi figure repreent the baeline ytem of the CASE 2016 Challenge, the propoe metho (n = 5, m = 100 m) ubmitte to the CASE 2016 Challenge (note that the feature are irreveribly compree) an the optimize metho (n = 3, m = 500 m), repectively. The claification accuracy of the optimize metho (n = 3, m = 500 m) wa improve by 9.77% compare with the baeline ytem. Thi illutrate the effect of uing both NN-GMM an concatenate acoutic feature. The bet claification accuracy of our metho (n = 3, m = 500 m) in the ten ranom ee wa 88.21% an wa improve by 2.61% compare with that of the propoe metho (n = 5 an m = 100 m), an wa ranke thir among the 49 algorithm.

5 Proceeing of APSIPA Annual Summit an Conference 2017 ata in both oun cene are imilar. Therefore, improvement of the claification accuracy for thee oun cene i require in the future. IV. CONCLUSION In thi paper, we optimize the number of concatenate frame an the frame concatenation interval for the previouly propoe metho. e carrie out an experiment uing the evelopment ataet an the evaluation ataet of the CASE 2016 Challenge. It wa foun that by etting the number of concatenate frame n to 5 or le an the frame concatenation interval m to 500 m or le, high claification accuracy wa obtaine relatively tably. Alo, the claification accuracy of oun cene having large temporal change wa improve. REFERENCES Fig. 5. Confuion matrix of the optimize metho (n = 3, m = 500 m), which i generate by counting the claification reult obtaine for all the ten ranom ee. Finally, Table IV how the claification accuracy for each oun cene. The column an row how the oun cene an the metho, repectively. Our metho (n = 3, m = 500 m) i greatly improve for grocery tore, cafe/retaurant an train compare with the previouly propoe metho (n = 1). The oun in grocery tore, cafe/retaurant an train are lou an untationary with large temporal change, incluing muic an peech. Figure 5 i the confuion matrix of the optimize metho (n = 3, m = 500 m), which i generate by counting the claification reult obtaine for all the ten ranom ee. The column an row how the target oun cene an the claifie reult, repectively. From Figure 5, many claification error for cafe/retaurant, train an library can be oberve. For example, the oun ata in cafe/retaurant are often incorrectly claifie a grocery tore, ince the oun [1]. Giannouli, E. Beneto,. Stowell, M. Roignol, M. Lagrange, M. Plumbley, etection an claification of acoutic cene an event: An IEEE AASP challenge, Proc. ASPAA 2013, Oct [2] G. Roma,. Nogueira, P. Herrera, Recurrence quantification analyi feature for auitory cene claification, Proc. ASPAA 2013, Oct [3] J. Nam Z. Hyung, K. Lee, Acoutic cene claification uing pare feature learning an elective max-pool by event etection, Proc. ASPAA 2013, Oct [4] G. E. Hinton, L. eng,. Yu, G. E. ahl, A. Mohame, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. N. Sainath, B. Kingbury, eep neural network for acoutic moeling in peech recognition: The hare view of four reearch group, IEEE Signal Proceing Magazine, Vol. 29, No. 6, pp , [5] G. Takahahi, T. Yamaa, S. Makino, N. Ono, Acoutic cene claification uing eep neural network an frame-concatenate acoutic feature, CASE 2016 Challenge Technical Report, Sep. 2016, technical report/tak1/takahahi 2016 tak1.pf. [6] [7] G. E. Hinton, A practical guie to training retricte Boltzmann machine, Momentum, Vol. 9, No. 1, [8] [9]