ENHANCING TIMBRE MODEL USING MFCC AND ITS TIME DERIVATIVES FOR MUSIC SIMILARITY ESTIMATION

Transcription

1 th Euroean Signal Processing Conference (EUSIPCO ) Bucharest, Romania, August 7-3, ENHANCING TIMBRE MODEL USING AND ITS TIME DERIVATIVES FOR MUSIC SIMILARITY ESTIMATION Franz de Leon, Kirk Martinez Electronics and Comuter Science, University of Southamton University Road, Southamton, United Kingdom SO7 BJ {fadld9, km}@ecs.soton.ac.uk ABSTRACT One of the oular methods for content-based music similarity estimation is to model timbre with as a single multivariate Gaussian with full covariance matrix, then use symmetric Kullback-Leibler divergence. From the field of seech recognition, we roose to use the same aroach on the s time derivatives to enhance the timbre model. The Gaussian models for the delta and acceleration coefficients are used to create their resective distance matrix. The distance matrices are then combined linearly to form a full distance matrix for music similarity estimation. In our exeriments on two datasets, our novel aroach erforms better than using alone. Moreover, erforming genre classification using k-nn showed that the accuracies obtained are already close to the state-of-the-art. Index Terms, music similarity estimation. INTRODUCTION Digital technology and the Internet have changed the music industry landscae. The increased accessibility of music has allowed consumers to store and share thousands of files on their comuter s hard disk, ortable media layer, mobile hone and other devices. Given the large music collections available, there is a need for new alications for browsing, organising, discovering as well as generating laylists for users. The research field of Music Information Retrieval (MIR) aims to address these challenges by using contentbased techniues for erforming tasks such as audio music similarity estimation and genre classification. Generally, the essential music dimension used in content-based aroaches is timbre. Timbre can be defined as the character or uality of a musical sound or voice as distinct from its itch and intensity []. It deends on the ercetion of the uality of sounds, which is related to the used musical instruments, with ossible audio effects, and to the laying techniues []. In the field of seech recognition, the mel-freuency cestral coefficients (s) have been widely used to model imortant characteristics in seech [3]. Since modelling seech characteristics and timbre are similar, the use of s has been extended with success in the field of music similarity [4]. In this aer, we roose to enhance s erformance in audio similarity tasks by using its time derivatives (e.g. delta and acceleration coefficients). The aer is organized as follows. The following section resents some related works. Section 3 details how the s, delta and acceleration coefficients are comuted and modelled. In section 4, we describe how the derived features are combined and used for audio music similarity estimation. The erformance is evaluated with varied arameters and the results are exlained in Section. Finally, we summarize our findings in Section 6.. RELATED WORK One of the standard aroaches to comute music similarity is to estimate a single multivariate Gaussian model on the vectors. In this way, the closed form solutions of the Kullback-Leibler (KL) divergence can be used to comute the similarity between two music models. Besides being fast, this method has been roven to outerform other more comlex music similarity aroaches []. Our aroach considers the time derivatives of the vectors to enhance music similarity estimation erformance. The time derivatives add dynamic information to the static cestral features [6]. In other studies, the delta and acceleration coefficients were aended to the static cestral features resulting in a three-fold increase in dimension (e.g. [9:9 :9 ]) [7]. These set of features can be used for genre classifiers. However, this would be imractical for uantifying music similarity estimation since the KL divergence involves numerically sensitive oerations and comutationally intensive matrix inversion. Our novel aroach simlifies the roblem by creating searate models for the time derivatives. Similar to the standard aroach to s, the time derivative vectors are summarized with a single multivariate Gaussian model. The same distance comutation is erformed for the resulting Gaussian models. The results are then combined with the original distances. EURASIP, - ISSN 76-46

2 3. MODELLING TIMBRE This section details the comutation of s and its time derivatives to model timbre. 3.. Mel-freuency cestral coefficients The timbre comonent is reresented by the s [3]. The normalized audio signals signal is divided into frames with a window size and ho size of samles (~3 msec.). The length of the segment ensures that the segmented signal is seudo-stationary while the ho size kees the continuity of the segments. Next, a window function (e.g. Hanning window) is alied to each segment. This is necessary to reduce sectral leakage. The following stes are then erformed to each segment:. Calculate the ower sectrum using FFT.. Transform the ower sectrum to Mel-scale using a filter bank consisting of triangular filters. 3. Get the sum of the freuency contents of each band. 4. Take the logarithm of each sum.. Comute the discrete cosine transform (DCT) of the logarithms. 3.. Delta and acceleration coefficients The erformance of a seech recognition system can be greatly enhanced by adding time derivatives to the basic static arameters [8]. Delta Coefficients are comuted using the following formula: d = t Θ θ ( c = + θ t θ ct θ Θ θ θ = where d t is a delta coefficient at time t, c is the cestral coefficient, comuted using a time window Θ. The same euation can be alied to the delta coefficients to obtain the acceleration coefficients. Figure visualizes the derived features for an audio cli Summarizing audio features The features derived from each audio track must be summarized efficiently and take into consideration the similarity comutation method that will be erformed. In this work, the Mel-freuency cestral coefficients are comuted for each time segment or frame. These features are aggregated using the bag-of-frames aroach to model global statistics. The bag-of-frames aroach is more aroriate in this case since the tasks that will be erformed are less selective, e.g. music similarity estimation. Previous works model the sectral information with a single Gaussian distribution with a diagonal covariance matrix [9]. Other studies used Gaussian Mixture ) () x audio cli Time (s) Delta Coefficients Acceleration Coefficients Frame Fig.. s, delta and acceleration coefficients for a - second audio cli. s to model the distributions using the K-means algorithm and exectation-maximization algorithm [], []. Subseuent works have shown that the same level of erformance can be achieved using single Gaussian distribution with full covariance matrix [], []. In this aer, the single Gaussian with full covariance aroach is imlemented to benefit from reduced comutational comlexity comared to the Gaussian Mixture s. We extend this aroach to delta and acceleration coefficients. Thus, each audio file is reresented by three single multivariate Gaussian models; for s, delta and acceleration coefficients. 4. APPLICATION TO AUDIO MUSIC SIMILARITY ESTIMATION In this section, we resent a music similarity estimation method using the derived features. Each audio file is reresented by three single multivariate Gaussian models. A single multivariate Gaussian robability density function is defined as: Ν(, ) = / T () x µ ex ( x µ ) ( x µ ) π where x is the observation (n-dimensional feature vector), µ is the mean, is an n x n covariance matrix. Using a single Gaussian with full covariance matrix to model a music file, the similarity between two tracks can be comuted using the Kullback-Leibler (KL) divergence. The KL divergence between two single Gaussians (x)=n(x;µ, ) and (x)=n(x;µ, ) is given by [3]: 6

3 A Song A TABLE I GENRE DISTRIBUTION OF TRACKS FOR THE ISMIR 4 AND DATASETS Song B log( + SKLD) log( + SKLD) log( + SKLD) Fig.. Block diagram of our roosed music similarity estimation system. KL( ) = log + ( µ µ ) A M M M T + Tr ( ) ( µ µ ) d where denotes the determinant of the matrix, Tr( ) denotes the trace function of a matrix. A common aroach to comute acoustic timbre similarity is to use the symmetric version of the Kullback- Leibler Divergence (SKLD), defined between two single Gaussian distributions x ~N(µ, ) and x ~N(µ, ): w w w3 Full + (3) SKLD ( x, x) = KL( x, x) + KL( x, x) (4) The main drawback of using these models is the fact that the SKLD does not hold the triangle ineuality and conseuently, is not a metric. It was shown that the transformation function T:SKLD {log(+skld)} / turns the symmetric Kullback-Leibler divergence into an exact metric when the statistical models comared are Gaussian [4]. To uantify music similarity based on the resented features, we comute air wise similarities using the transformed symmetric KL divergence. This ste roduces three distance matrices. For each distance matrix, we aly distance sace normalization []. Finally, the distance matrices are linearly combined into a full distance matrix. The weights of the linear combination must be otimized for the roosed system. For a given feature set, the weights are in the range of to with a ste size of.. The sum of the weights is. An intuitive aroach is used instead of erforming exhaustive comarisons by comuting all ossible ermutations. ISMIR 4 Training Set Songs 79 Genres classical (3), electronic (), jazz_blues (6), metal_unk (4), rock_o (), world () Full Set (Training & Develoment) Songs 48 Genres classical (64), electronic (9), jazz_blues (), metal_unk (9), rock_o (3), world (44) Songs Genres country (), rock (), reggae (), blues (), disco (), hiho (), jazz (), o (), classical (), metal ().. Setu. EXPERIMENTS Two datasets were used in our exeriments. The first dataset is the training and develoment sets for the ISMIR 4 genre classification contest [6]. Both the training and testing set are comosed of tracks from six genres. The second dataset is the genre collection [7]. The dataset consists of audio each 3 seconds long. It contains genres, each reresented by tracks. For each track, a 3-second cli was selected from the middle. For files that are less than 3 sec. long, the actual length was used. Each signal was normalized then divided into short overlaing segments (e.g. 3ms). The s were derived using 36 filter banks on Hanning-windowed segments. Twenty cestral coefficients were obtained but only the last 9 were used. The delta and acceleration coefficients were then derived using Euation. Objective statistics were derived from the full distance matrix. In music information retrieval, music similarity is taken in the context of genre, artist or album similarity. For our tests, the metric we used was the ercentage of genre matches in the to,, and uery (recision at R =,,, ). { relevant items} { retrieved items} recision = { retrieved items} Artist filtering was alied for the ISMIR 4 training set for comarison. This means that there is only one track er artist in the artist filtered dataset. The exeriments were erformed using different time windows for the delta coefficients, and weights for the individual distance matrices. Finally, we evaluated genre classification accuracy for the two datasets to comare the erformance of combining with its time derivatives. () 7

4 .. Results In Table II, we tabulate the recision after returning R items using the otimum weights. Note that these recisions are resented as the average across all the genres for the articular dataset. Moreover, only the best combinations are resented. Based on the results, there is a significant imrovement in the recision in using in conjunction with the delta coefficients than using alone. This roves the imortance of time derivatives as the static s alone don t have temoral information. On the average, the best recisions were obtained when the delta coefficients were given more weight than (w =.4, w =.6). However, this does not imly that delta coefficients could relace s to model timbre. For examle, using delta coefficients alone on dataset we obtained - recision of.738; while for alone, There were no significant imrovements or degradation in the recisions using three features (.9*+. ) as comared to using only two (*.4+.6 ). This means that the acceleration coefficients can be disregarded in modeling timbre. Thus, the exeriments showed that a good model for timbre involves the s and its delta coefficients. The exeriments also determined if the window size used to comute the time derivatives can affect the system s erformance. Table II shows inconsistency in the results. However in most cases, using Θ=3 frames is better than Θ= frames. Using the ISMIR 4 dataset, we comared the recision with and without artist filtering. With artist filtering, the number of returned items was limited to since there are only tracks in the jazz_blues genre. Without artist filtering, the recision is around.73; as comared to with artist filtering that resulted to around.. There is a difference in the erformance of around. which is already consistent with other studies that used the same dataset [8]. Our final exeriments used k-nearest neighbors to erform genre classification. This method, while being simle, is already established for erforming music similarity measures [9][]. We want to determine the imrovement in the classification accuracy using our roosed timbre model. Figure 3 shows that the combination of and its time derivatives consistently erform better than alone. For the ISMIR 4 and datasets, the best accuracies at k= are.8 and.86 resectively (.4+.6 ). As reviously observed, there is no significant difference in the accuracies using three features. Using only these simle features that model timbre, it is worthy to note that the erformance of the system is not far from the state-of-the-art. Thus, the new timbre model can serve as a foundation that can be enhanced by other features (e.g. fluctuation atterns [], onset atterns [9], temo []). TABLE II R-PRECISION USING, DELTA AND ACCELERATION COEFFICIENTS Collection Features Artist Returned Items, R Filter ISMIR4 no Train, Θ=3).4+.6 no *+. no yes.466 na na na.4+.6 yes.9 na na na.9*+. yes.37 na na na ISMIR4 Train, Θ=) ISMIR4 Full, (Θ=3) ISMIR4 Full, (Θ=) (Θ=3) (Θ=).4+.6 no *+. no yes.43 na na na.9*+. yes.7 na na na no no *+. no no *+. no no no *+. no no *+. no In terms of comutational comlexity, there is not much overhead added in comuting the symmetric Kullback-Leibler divergence twice (e.g. and delta coefficients). The system works on stored matrices such as the covariance and inverse covariance matrices. The results show that using timbre models is imortant for content-based music similarity estimation. Timbre may contain salient information that roughly describes music genre. Research had shown that humans have the ability to distinguish and classify music after listening to short clis of audio. This imlies the viability of using timbre as this feature can be easily extracted from short clis. The major limitation is that humans do not comute a weighted sum of similarities with resect to different asects of music. In fact, the concet of audio similarity is subjective to listeners and a single asect which is similar can be considered to judge similarity. Nevertheless, the comutational model resented in this aer hoes to contribute on the imrovement of content-based systems. 6. CONCLUSION We have investigated a method for enhancing features for music similarity estimation using its time derivatives. Our novel aroach alies the standard single multivariate Gaussian with full covariance matrix to model s delta and acceleration coefficients. Using the Kullback-Leibler divergence to calculate music similarity, exeriments have shown a consistent imrovement in the erformance in using the delta coefficients in conjunction with the s. Performing genre classification using k- NN showed that the accuracies obtained are already close to the state-of-the-art. In addition, recent studies have roved that our aroach has a otential to be alied on larger databases [4][]. We will further investigate our method and determine how other low-level features can be used to imrove these initial results. We will also exlore alternative ways of integrating these low-level features with our timbre model. 8

5 accuracy accuracy d.9*+.dd random baseline Fig. 3. Genre classification accuracy for ISMIR 4 (to) and (bottom) datasets. 7. ACKNOWLEDGMENTS Mr. Franz de Leon is suorted by the Engineering Research and Develoment for Technology Faculty Develoment Program of the University of the Philiines and DOST. 8. REFERENCES [] B. C. J. Moore, An Introduction to the Psychology of Hearing. Academic Press, 3,. 43. [] N. Orio, Music Retrieval: A Tutorial and Review, Foundations and Trends in Information Retrieval, vol., no.,. -96, 6. [3] S. Davis and P. Mermelstein, Comarison of arametric reresentations for monosyllabic word recognition in continuously soken sentences, IEEE Transactions on Acoustics, Seech, and Signal Processing, vol. 8, no. 4, , Aug. 98. [4] M. A. Casey, R. Veltkam, M. Goto, M. Leman, C. Rhodes, and M. Slaney, Content-Based Music Information Retrieval: Current Directions and Future Challenges, Proceedings of the IEEE, vol. 96, no. 4, , Ar. 8. [] M. I. Mandel and D. P. W. Ellis, Song-level Features and Suort Vector Machines for Music Classification, in Submission to MIREX,, [6] S. Furui, Seaker-indeendent isolated word recognition based on emhasized sectral dynamics, in ICASSP 86. k.4+.6d.9*+.dd random baseline k IEEE International Conference on Acoustics, Seech, and Signal Processing, 99, vol., [7] D. Turnbull, L. Barrington, D. Torres, and G. Lanckriet, Semantic Annotation and Retrieval of Music and Sound Effects, IEEE Transactions on Audio, Seech, and Language Processing, vol. 6, no., , Feb. 8. [8] J.-julien Aucouturier and F. Pachet, Imroving Timbre Similarity : How high s the sky?, J. Negative Results Seech Audio Sci., vol., 4. [9] G. Tzanetakis, G. Essl, and P. Cook, Automatic Musical Genre Classification Of Audio Signals, in Proceedings of ISMIR : The International Conference on Music Information Retrieval and Related Activities,. [] B. Logan and A. Salomon, A music similarity function based on signal analysis, in IEEE International Conference on Multimedia and Exo,. ICME.,, [] J.-julien Aucouturier and F. Pachet, Music Similarity Measures : What s the Use?, in 3rd International Conference on Music Information Retrieval,, [] E. Pamalk, Audio-Based Music Similarity and Retrieval : Combining a Sectral Similarity with Information Extracted from Fluctuation Patterns, in Submission to MIREX 6, 6. [3] W. Penny, Kullback-Liebler Divergences of Normal, Gamma, Dirichlet and Wishart Densities,. [4] C. Charbuillet, G. Peeters, S. Barton, and V. Gouet-Brunet, A fast algorithm for music search by similarity in large databases based on modified Symetrized Kullback Leibler Divergence, in International Worksho on Content Based Multimedia Indexing (CBMI),,. -6. [] K. Seyerlehner, M. Schedl, T. Pohle, and P. Knees, Using Block-Level Features for Genre Classification, Tag Classification and Music Similarity Estimation, in Submission to Audio Music Similarity and Retrieval Task of MIREX,. [6] ISMIR 4 Audio Descrition Contest-Genre/Artist ID Classification and Artist Similarity. [Online]. Available: htt://ismir4.ismir.net/genre_contest/index.htm. [7] G. Tzanetakis and P. Cook, Musical genre classification of audio signals, IEEE Transactions on Seech and Audio Processing, vol., no.,. 93-3, Jul.. [8] J. H. Jensen, M. G. Christensen, M. N. Murthi, and S. H. Jensen, Evaluation of Estimation Techniues for Music Similarity, in Proceedings of the 4th Euroean Signal Processing Conference, 6, no. Eusico. [9] T. Pohle, D. Schnitzer, M. Schedl, P. Knees, and G. Widmer, On rhythm and general music similarity, in Submission to Audio Music Similarity and Retrieval Task of MIREX 9, 9, no. Ismir,. -3. [] E. Pamalk, Comutational s of Music Similarity and their Alication in Music Information Retrieval, Vienna University of Technology, 6. [] F. D. Leon and K. Martinez, Submission to MIREX Genre Classification and and Audio Similarity Tasks, in Submission to Audio Music Similarity and Retrieval Task of MIREX,. [] D. Schnitzer, A. Flexer, and G. Widmer, A Filter-and- Refine Method for Fast Similarity Search in Millions of Tracks, in ISMIR 9, 9, no. Aril,