Session 1: Pattern Recognition

Transcription

1 Proc. Digital del Continguts Musicals Session 1: Pattern Recognition Music Content Analysis Pattern Classification The Statistical Approach Distribution Models Singing Detection Dan Ellis <dpwe@ee.columbia.edu> Laboratory for Recognition and Organization of Speech and Audio Columbia University, New York Spring 2003 Musical Content - Dan Ellis 1: Pattern Recognition

2 1 Music Content Analysis Music contains information at many levels - what is it? We d like to get this information out automatically - fine-level transcription of events - broad-level classification of pieces Information extraction can be framed as: pattern classification / recognition or machine learning - build systems based on (labeled) training data Musical Content - Dan Ellis 1: Pattern Recognition

3 Music analysis What might we want to get out of music? Instrument identification - different levels of specificity - registers within instruments Score recovery - transcribe the note sequence - extract the performance Ensemble performance - gestalts : chords, tone colors Broader timescales - phrasing & musical structure - artist / genre clustering and classification Musical Content - Dan Ellis 1: Pattern Recognition

4 Course Outline Session 1: Pattern Classification - MLP Neural Networks - GMM distribution models - Singing detection Session 2: Sequence Recognition - The Hidden Markov Model - Chord sequence recognition Session 3: Musical Applications - Note transcription - Music summarization - Music Information Retrieval - Similarity browsing Musical Content - Dan Ellis 1: Pattern Recognition

5 Outline Music Content Analysis Pattern Classification - classification - decision boundaries - neural networks The Statistical Approach Distribution Models Singing Detection Musical Content - Dan Ellis 1: Pattern Recognition

6 2 Pattern Classification Classification means: finding categorical (discrete) labels for real-world (continuous) observations F2/Hz 4000 f/hz time/s x x F1/Hz ay ao Why? - information extraction - conditional processing - detect exceptions Musical Content - Dan Ellis 1: Pattern Recognition

7 Building a classifier Define classes/attributes - could state explicit rules - better to define through training examples Define feature space Define decision algorithm - set parameters from examples Measure performance - calculate (weighted) error rate 1100 Pols vowel formants: "u" (x) vs. "o" (o) 1000 F2 / Hz F1 / Hz Musical Content - Dan Ellis 1: Pattern Recognition

8 Classification system parts Sensor signal segment feature vector Pre-processing/ segmentation Feature extraction STFT Locate vowels Formant extraction class Classification Post-processing Context constraints Costs/risk Musical Content - Dan Ellis 1: Pattern Recognition

9 Feature extraction Right features are critical - waveform vs. formants vs. cepstra - invariance under irrelevant modifications Theoretically equivalent features may act very differently in practice - representations make important aspects explicit - remove irrelevant information Feature design incorporates domain knowledge - more data less need for cleverness? Smaller feature space (fewer dimensions) simpler models (fewer parameters) less training data needed faster training Musical Content - Dan Ellis 1: Pattern Recognition

10 Minimum distance classification Pols vowel formants: "u" (x), "o" (o), "a" (+) F2 / Hz x * o new observation x F1 / Hz Find closest match (nearest neighbor) min[ D( x, y ω i )], - choice of distance metric D(x,y) is important Find representative match (class prototypes) min[ D( x, z ω )] - class data { y ω,i } class model z ω Musical Content - Dan Ellis 1: Pattern Recognition

11 Linear classifiers Minimum Euclidean distance is equivalent to a linear discriminant function: - examples { y ω,i } template z ω = mean{ y ω,i } - observed feature vector x = [x 1 x 2...] T - Euclidean distance metric Dxy [, ] = ( x y) T ( x y) - minimum distance rule: class i = argmin i D[ x, z i ] = argmin i D 2 [ x, z i ] - i.e. choose class O over U if D 2 [ x, z O ] < D 2 [ x, z U ]... Musical Content - Dan Ellis 1: Pattern Recognition

12 Linear classification boundaries Min-distance divides normal to class centers: decision boundary 1 (z O -z U ) T (z O -z U ) 2 z O -z U z U x o z O (z O -z U ) (x-z U ) T (z O -z U ) z O -z U * x Scaling axes changes boundary: new boundary x o Squeezed horizontally x Squeezed vertically new boundary o Musical Content - Dan Ellis 1: Pattern Recognition

13 Decision boundaries Linear functions give linear boundaries - planes and hyperplanes as dimensions increase Linear boundaries can become curves under feature transformations - e.g. a linear discriminant in x' = [x 2 1 x ] T What about more complex boundaries? - i.e. if a linear discriminant is how about a nonlinear function? F[ w i x i ] F[ w ij x i ] w i x i - changes only threshold space, but j i j i F[ w jk F[ w ij x i ]] offers more flexibility, and has even more... Musical Content - Dan Ellis 1: Pattern Recognition

14 Neural networks Sums over nonlinear functions of sums large range of decision surfaces e.g. Multi-layer perceptron (MLP) with 1 hidden layer: y k = F[ w jk F[ w ij x j i ]] j x 1 + x w 2 h jk + y 1 x F[ ] Input layer w ij + h 1 Hidden layer Problem is finding the weights w ij... (training) + y2 Output layer Musical Content - Dan Ellis 1: Pattern Recognition

15 Back-propagation training Find w ij to minimize e.g. E = n for training set of patterns {x n } and desired (target) outputs {t n } y( x n ) t n Differentiate error with respect to weights - contribution from each pattern x n, t n : y k = F[ w jk h j j ] E w jk = = E y y Σ w jk 2( y t) F' Σ w jk = w jk α j ( w jk h j ) [ ] h j E w jk i.e. gradient descent with learning rate α 2 Musical Content - Dan Ellis 1: Pattern Recognition

16 Neural net example 2 input units (normalized F1, F2) 5 hidden units, 3 output units ( U, O, A ) F1 F2 A O U Sigmoid nonlinearity: F[ x] 1 = e x df = F( 1 F) dx sigm(x) d sigm(x) dx Musical Content - Dan Ellis 1: Pattern Recognition

17 1 Neural net training 2:5:3 net: MS error by training epoch Mean squared error Training epoch 10 iterations iterations example Musical Content - Dan Ellis 1: Pattern Recognition

18 Outline Music Content Analysis Pattern Classification The Statistical Approach - Conditional distributions - Bayes rule Distribution Models Singing Detection Musical Content - Dan Ellis 1: Pattern Recognition

19 3 The Statistical Approach Observations are random variables whose distribution depends on the class: Observation x Class ω i (hidden) discrete continuous Source distributions p(x ω i ) - reflect variability in feature - reflect noise in observation - generally have to be estimated from data (rather than known in advance) p(x ω i ) p(x ω i ) Pr(ω i x) ω 1 ω 2 ω 3 ω 4 x Musical Content - Dan Ellis 1: Pattern Recognition

20 Random Variables review Random vars have joint distributions (pdf s): p(x,y) y µy σ y σ xy p(y) p(x) µ x σ x x - marginal px ( ) = pxy (, ) dy - covariance Σ = E[ ( o µ )( o µ ) T ] = σ x 2 σxy σ xy σ y 2 Musical Content - Dan Ellis 1: Pattern Recognition

21 Conditional probability Knowing one value in a joint distribution constrains the remainder: y Y p(x,y) x p(x y = Y ) Bayes rule: pxy (, ) = pxy ( ) py ( ) pyx ( ) = pxy ( ) py ( ) px ( ) can reverse conditioning given priors/marginal - either term can be discrete or continuous Musical Content - Dan Ellis 1: Pattern Recognition

22 Priors and posteriors Bayesian inference can be interpreted as updating prior beliefs with new information, x: Bayes Rule: Pr( ω i ) Prior probability pxω ( i ) pxω ( j ) Pr( ω j ) j Likelihood Evidence = p(x) = Pr( ω i x) Posterior probability Posterior is prior scaled by likelihood & normalized by evidence (so Σ(posteriors) = 1) Objection: priors are often unknown - but omitting them amounts to assuming they are all equal Musical Content - Dan Ellis 1: Pattern Recognition

23 Bayesian (MAP) classifier Minimize the probability of error by choosing maximum a posteriori (MAP) class: Intuitively right - choose most probable class in light of the observation Given models for each distribution, hence ωˆ = argmax ω i Pr( ω i x) pxω ( i ) the search for ωˆ = argmax Pr( ω i x) ω i pxω ( i ) Pr( ω i ) becomes argmax ω i pxω ( j ) Pr( ω j ) j but denominator = p(x) is the same over all ω i ωˆ = argmax ω i pxω ( i ) Pr( ω i ) Musical Content - Dan Ellis 1: Pattern Recognition

24 Practical implementation Optimal classifier is but we don t know ωˆ = Pr( ω i x) argmax ω i Pr( ω i x) Can model directly e.g. train a neural net to map from inputs x to a set of outputs Pr(ω i ) - a discriminative model Often easier to model conditional distributions then use Bayes rule to find MAP class pxω ( i ) Labeled training examples {x n,ω xn } Sort according to class Estimate conditional pdf for class ω 1 p(x ω 1 ) Musical Content - Dan Ellis 1: Pattern Recognition

25 Outline Music Content Analysis Pattern Classification The Statistical Approach Distribution Models - Gaussian models - Gaussian mixtures Singing Detection Musical Content - Dan Ellis 1: Pattern Recognition

26 4 Distribution Models Easiest way to model distributions is via parametric model - assume known form, estimate a few parameters Gaussian model is simple & useful: in 1 dim: pxω ( i ) = πσ i exp normalization to make it sum to x µ i σ i Parameters mean µ i and variance σ i 2 fit P M P M e 1/2 p(x ω i ) µ i σ i x Musical Content - Dan Ellis 1: Pattern Recognition

27 Gaussians in d dimensions: p( x ω i ) = ( 2π) d 1 2 Σ i exp 1 -- ( x µ 2 i ) T 1 Σ i ( x µ i ) Described by d dimensional mean vector µ i and d x d covariance matrix Σ i argmax ω i Classify by maximizing log likelihood i.e ( x µ 2 i ) T 1 Σ i ( x µ i ) 1 -- log Σ 2 i + logpr( ω i ) Musical Content - Dan Ellis 1: Pattern Recognition

28 Gaussian Mixture models (GMMs) Single Gaussians cannot model - distributions with multiple modes - distributions with nonlinear correlation What about a weighted sum? px ( ) c k pxm ( k ) i.e. k where {c k } is a set of weights and pxm ( k ) { } is a set of Gaussian components - can fit anything given enough components Interpretation: each observation is generated by one of the Gaussians, chosen at random with priors c k = Pr( m k ) Musical Content - Dan Ellis 1: Pattern Recognition

29 Gaussian mixtures (2) e.g. nonlinear correlation: resulting surface original data Gaussian components Problem: finding c k and m k parameters - easy if we knew which m k generated each x Musical Content - Dan Ellis 1: Pattern Recognition

30 Expectation-maximization (EM) General procedure for estimating a model when some parameters are unknown - e.g. which component generated a value in a Gaussian mixture model Procedure: Iteratively update fixed model parameters Θ to maximize Q=expected value of log-probability of known training data x trn and unknown parameters u: Q( Θ, Θ old ) = Pr u x trn, Θ old ( ) log p( u, x trn Θ) u - iterative because Pr( u x trn, Θ old ) changes each time we change Θ - can prove this increases data likelihood, hence maximum-likelihood (ML) model - local optimum - depends on initialization Musical Content - Dan Ellis 1: Pattern Recognition

31 Fitting GMMs with EM Want to find component Gaussians m k { µ k, Σ k } plus weights = = c k Pr( m k ) to maximize likelihood of training data x trn If we could assign each x to a particular m k, estimation would be direct Hence, treat ks (mixture indices) as unknown, form Q = E[ log p( x, k Θ) ], differentiate with respect to model parameters equations for µ k, Σ k and c k to maximize Q... Musical Content - Dan Ellis 1: Pattern Recognition

32 GMM EM update equations: Solve for maximizing Q: pkx n, Θ n µ k = pkx ( n, Θ) n pkx n, Θ n Σ k = n ( ) x n ( )( x n µ k )( x n µ k ) T pkx ( n, Θ) 1 c k = --- pkx N ( n, Θ) n pkx ( n, Θ) Each involves, fuzzy membership of x n to Gaussian k Parameter is just sample average, weighted by fuzzy membership Musical Content - Dan Ellis 1: Pattern Recognition

33 GMM examples Vowel data fit with different mixture counts: 1 Gauss logp(x)= Gauss logp(x)= Gauss logp(x)= Gauss logp(x)= example... Musical Content - Dan Ellis 1: Pattern Recognition

34 Methodological Remarks: Training and test data A rich model can learn every training example (overtraining) error rate Test data Training data Overfitting training or parameters But, goal is to classify new, unseen data i.e. generalization - sometimes use cross validation set to decide when to stop training For evaluation results to be meaningful: - don t test with training data! - don t train on test data (even indirectly...) Musical Content - Dan Ellis 1: Pattern Recognition

35 Model complexity More model parameters better fit - more Gaussian mixture components - more MLP hidden units More training data can use larger models For best generalization (no overfitting), there will be some optimal model size: More parameters Word Error Rate More training data WER% Optimal parameter/data ratio Constant training time Training set / hours Hidden layer / units WER for PLP12N-8k nets vs. net size & training data Musical Content - Dan Ellis 1: Pattern Recognition

36 Outline Music Content Analysis Pattern Classification The Statistical Approach Distribution Models Singing Detection - Motivation - Features - Classifiers Musical Content - Dan Ellis 1: Pattern Recognition

37 5 Singing Detection (Berenzweig et al. 01) Hz File: /Users/dpwe/projects/aclass/aimee.wav t Can we automatically detect when singing is present? f 9 Printed: Tue Mar 11 13:04:28 0:02 0:04 0:06 0:08 0:10 0:12 0:14 0:16 0:18 0:20 0:22 0:24 0:26 0:28 mus vox mus vox mus - for further processing (lyrics recognition?) - as a song signature? - as a basis for classification? Musical Content - Dan Ellis 1: Pattern Recognition

38 Singing Detection: Requirements Labelled training examples - 60 x 15 sec. radio excerpts - hand-mark sung phrases freq / khz freq / khz trn/mus/ hand-label vox 0 trn/mus/ Labelled test data - several complete tracks from CDs, hand-labelled Feature choice - Mel-frequency Cepstral Coefficients (MFCCs) popular for speech; maybe sung voice too? - separation of voices? - temporal dimension freq / khz freq / khz trn/mus/19 trn/mus/ time / sec Classifier choice - MLP Neural Net - GMMs for singing / music - SVM? Musical Content - Dan Ellis 1: Pattern Recognition

39 MLP Neural Net Directly estimate p(singing x) music MFCC calculation C 0 C 1... singing not singing C 12 - net has 26 inputs (+ ), 15 HUs, 2 o/ps (26:15:2) How many hidden units? - depends on data amount, boundary complexity Feature context window? - useful in speech Delta features? - useful in speech Training parameters... Musical Content - Dan Ellis 1: Pattern Recognition

40 MLP Results Raw net outputs on a CD track (FA 74.1%): Aimee Mann : Build That Wall + handvox freq / khz :15:1 netlab on 16ms frames thr=0.5) p(singing) time / sec 30 p(singing) Smoothed for continuity: best FA = 90.5% y p ( ) ( ) time / sec 30 Musical Content - Dan Ellis 1: Pattern Recognition

41 GMM System Separate models for p(x sing), p(x no sing) - combined via likelihood ratio test GMM1 p(x singing ) music MFCC calculation C 0 C 1... Log l'hood ratio test p(x singing ) log p(x not ) singing? C 12 GMM2 p(x no singing ) How many Gaussians for each? - say 20; depends on data & complexity What kind of covariance? - diagonal (spherical?) Musical Content - Dan Ellis 1: Pattern Recognition

42 GMM Results Raw and smoothed results (Best FA=84.9%): freq / khz log(lhood) Aimee Mann : Build That Wall + handvox 26d 20mix GMM on 16ms frames thr=0) log(lhood) d 20mix GMM smoothed by 61 pt (1 sec) hann thr=-0.8) time / sec 30 MLP has advantage of discriminant training Each GMM trains only on data subset faster to train? (2 x 10 min vs. 20 min) Musical Content - Dan Ellis 1: Pattern Recognition

43 Summary Music content analysis: Pattern classification Basic machine learning methods: Neural Nets, GMMs Singing detection: classic application but... the time dimension? Musical Content - Dan Ellis 1: Pattern Recognition

44 References A.L. Berenzweig and D.P.W. Ellis (2001) Locating Singing Voice Segments within Music Signals, Proc. IEEE Workshop on Apps. of Sig. Proc. to Acous. and Audio, Mohonk NY, October R.O. Duda, P. Hart, R. Stork (2001) Pattern Classification, 2nd Ed. Wiley, E. Scheirer and M. Slaney (1997) Construction and evaluation of a robust multifeature speech/music discriminator, Proc. IEEE ICASSP, Munich, April Musical Content - Dan Ellis 1: Pattern Recognition