Sound event detection and rhythmic parsing of music signals

Size: px

Start display at page:

Download "Sound event detection and rhythmic parsing of music signals"

Alisha Paul
5 years ago
Views:

1 Soud evet detectio ad rhythmic parsig of music sigals ISMIR Graduate School, October 4th-9th, 2004 Cotets: Itroductio Measurig the degree of chage i music sigals Oset detectio Rhythmic pulse estimatio Higher-level modelig for rhythmic parsig Demostratios

2 Classificatio 2 Itroductio Oset detectio = Detectio of the begiigs of discrete acoustic evets i a acoustic sigals Rhythmic parsig (= musical meter aalysis) detectig momets of musical stress i a acoustic sigal ad processig them so that uderlyig periodicities are discovered e.g. tappig foot to music rhythmically meaigful segmetatio of musical sigals at differet time scales Applicatios temporal framework for audio editig audio/video sychroizatio music segmetatio for further aalysis (e.g. trascriptio)

3 2 Measurig the degree of chage i music Classificatio 3 Momets of chage are importat for oset detectio ad rhythmic parsig Percept of a oset is caused by a oticeable chage i the itesity, pitch or timbre of a soud momets of musical stress (accets) are caused by the begiigs of soud evets, sudde chages i loudess or timbre, harmoic chages Perceptual chage should be estimated to detect what humas detect ad to igore what humas igore (music is chagig all the time!) to do musically meaigful rhythmic parsig

4 Measurig the degree of chage i music Classificatio 4 Time-domai sigal some data reductio is eeded But: the power evelope of a sigal is ot sufficiet Power evelope Frequecy selectivity of hearig: audibility of a chage at each critical bad is oly affected by the spectral compoets withi the same bad compoets withi a sigle critical bad may mask each other but: if the frequecy separatio is sufficietly large, the maskig compoet must be about millio times louder tha the other Need to measure chage idepedetly at critical bads, ad the combie the results

5 Measurig the degree of chage i music Classificatio 5 Scheirer s classical demostratio: perceived rhythmic cotet of may music types remais the same if oly the power evelopes of a few subbads are preserved ad the used to modulate a white oise sigal oe bad is ot eough applies to music with strog beat Let s repeat the experimet: too much data reductio or ot? Bretwood Sambafrique The Bells Starig

6 Measurig degree of chage i music Badwise processig Classificatio 6 Perceived chage at subbad music sigal Filterbak Combie results output Filterbak: Fourier trasforms i successive 23ms time frames (50% overlap) i each frame, 36 triagular-respose badpass filters are simulated that are uiformly distributed o the Mel frequecy scale (50Hz 20kHz)

7 Measurig degree of chage i music Degree of chage at each bad Classificatio 7 Deote by x b () the power evelope at critical bad b=,...,36 as a fuctio of time (frame idex) How to measure the degree of chage at subbads? Differetial? For humas, the smallest detectable chage i itesity, I, is approximately proportioal to the itesity I of the sigal, the same amout of icrease beig more promiet i a quiet sigal. Audible ratio I / I is approximately costat Thus it is reasoable to ormalize the differetial of power with power: ( d / dt) x b x ( ) ( ) b = d dt l Figure (piao oset): dashed lie: (d/dt) x b () solid lie: (d/dt) l[x b ()] [ x ( ) ] b

Measurig degree of chage i music Degree of chage at each bad Classificatio 8 A umerically robust way of calculatig the logarithm is the µ-law compressio, y b ( ) = l [ + µ xb ( ) ] l( + µ ) costat µ

8 Measurig degree of chage i music Degree of chage at each bad Classificatio 8 A umerically robust way of calculatig the logarithm is the µ-law compressio, y b ( ) = l [ + µ xb ( ) ] l( + µ ) costat µ determies the degree of compressio for x b () (µ= / σ x ) Lowpass filter the compressed power evelopes at 0Hz deote resultig sigal with z b () Differetiate, ad retai oly positive chages (HWR(x)=max(x, 0)): z b () = HWR{z b () z b ( )} Form weighted sum of z b () ad z b (): u b () = ( λ) z b () + λ (f r /f LP ) z b () where λ=0.8 (or ) balaces betwee z b () ad z b (), ad (f r /f LP ) is a ormalizig costat

9 Measurig degree of chage i music Degree of chage at each bad Classificatio 9 Figure: illustratio of the dyamic compressio ad weighted differetiatio steps for a artificial subbad sigal x b () u b () x b ()

10 Measurig degree of chage i music Summary music sigal Filterbak x b ()... Perceived chage at subbad u b ()... Classificatio 0 Combie results output v() power evelope µ-law compress lowpass filter d / dt, rectify Fially: sum across chaels to estimate overall chage v( ) = 36 b= u b ( )

11 Measured chage sigals Classificatio Bretwood Jazz Quartet Lee Riteour Lyyrd Skyyrd v() v() v()

12 Measured chage sigals Classificatio 2 Bach Beethove v() v()

13 Classificatio 3 2 Oset detectio Adaptive thresholdig ad peak pickig Robust oe-by-oe detectio of osets is hard to attai! Bretwood: v() Beethove v() sigal level adaptatio would be eeded

14 Classificatio 4 Oset detectio Remark: there are several other approaches to oset detectio, too Methods based o complex-domai upredictability probabilistic modelig (model chage detectio) chages at the output of a auditory model (pitch icluded) idepedet compoet aalysis...

15 Classificatio 5 3 Musical meter Characterizes the temporal regularity of a music sigal Figure: Musical meter is hierarchical i structure pulse sesatios at differet time scales tactus level is the most promiet ( foot tappig rate ) tatum: time quatum (fastest pulse) measure pulse: related to harmoic chage rate

16 Meter aalysis Existig methods Classificatio 6 (Early work: Steedma77, LoguetHL82,84, LerdahlJ83, Lee85, PovelE85) System Iput Aim Approach Evaluatio material Parcutt 994 score meter rule-based artificial sythesized patters Brow 994 score meter autocorrelatio classical scores Rosethal 992 MIDI meter rule-based 92 piao performaces LargeK 994 MIDI meter oscillators few example pieces Temperley 999 MIDI meter, quat. rule-based source codes available Dixo 200 MIDI, audio tactus rule-based source codes available CemgilK MIDI tactus, quat. probabilistic model expressive performaces Raphael 200 MIDI, audio tactus, quat. probabilistic model expressive performaces GotoM audio meter DSP 85 pieces, pop, 4/4 time Scheirer 998 audio tactus DSP strog beat,sources available Laroche 200 audio tactus, swig probabilistic model steady-tempo music demos SetharesS 200 audio meter DSP steady-tempo music examples GouyoHC 2002 audio tatum pulse DSP 57 drum tracks, steady tempo 2003 audio meter DSP + probabilistic 474 pieces, all music types

17 Meter aalysis Overview of the TUT method Classificatio 7

18 Meter aalysis 3. Degree of accet (chage) Classificatio 8 Accet sigals (degree of chage) Degree of accetuatio as a fuctio of time at four frequecy chaels v c ( ) = cm u b b= ( c ) M + ( )

19 Meter aalysis 3.2 Metrical pulse salieces Classificatio 9 Metrical pulse salieces (weigths) Salieces of differet metrical pulses at time (resoator eergies)

20 Meter aalysis Bak of comb filter resoators Classificatio 20 Pulse salieces are obtaied by aalyzig the periodicity of the four accet sigals Comb filters were foud very suitable for this purpose origially proposed by Scheirer

21 Meter aalysis Comb filters Trasfer fuctio H a = az ( z) k x() -a a z -k Classificatio 2 y() Magitude respose a = 0.9 k = 7 Impulse respose:

22 Meter estimatio Bak of comb filter resoators Classificatio 22

23 Meter aalysis 3.3 Higher-level modelig Classificatio 23 Meter tatum, tactus, measure Fids pulse periods first ad the phases oly for the wiig periods

24 Pulse periods HMM Time Classificatio 24 Measure: Tactus: Tatum: Observatio: (comb filter eergies)

25 Meter aalysis Probabilistic model Classificatio 25 Values of the three hidde variables (pulse periods) are joitly defied usig a state variable q = [ j, k, l ] equivalet to A = j, B = k, C = l Hidde-state process is a first-order Markov process iitial state distributio P(q ) trasitio probabilities P(q q ) state-coditioal observatio desities P(s q ) Joit probability desity of a state sequece Q = (q q 2...q N ) ad observatio sequece O = (s s 2...s N ) : p N P( q q ) p( s ) = 2 ( Q O) = P( q ) P( s q), q A remaiig problem is to fid reasoable estimates for the model parameters

26 Meter aalysis Observatio likelihoods Classificatio 26 By makig a series of assumptios we arrive at the followig approximatio for P(s q ) : P(s q =[j, k, l]) = s ( k) s ( l) S ( /j) where S (/) is the Fourier trasform of s () By defiitio, other pulse periods are iteger multiples of the tatum period overall s(, ) gives iformatio about tatum B Observatio s Spectrum S

27 Meter aalysis Priors Classificatio 27 Period priors: 2-parameter log-ormal distributio suggested by Parcutt (994)

28 Classificatio 28 Meter aalysis Prior probability for meter Assumptios of our model ca be writte as Coditioal probabilities are preseted as a product Gaussia mixture desity f(x) models the relatio depedecies of simultaeous periods idepedet of their frequecies of occurrece realizes a tedecy towards biary ad terary iteger relatios ( ) ( ) ( ) ( ) C B C A B A B B P P P q q P,, = ( ) ( ) ( ) ( ) ( ) ( ) = B C C C B C C C B C C C C B C f P P P P P P,,

29 Classificatio 29 Meter aalysis Trasitio probabilities Trasitio probabilities modeled as a product of the a priori probability for a certai period ad a term which describes tedecy to remai i a certai period Fuctio g implemets Normal distributio as a fuctio of the relative chage i the period ( ) ( ) ( ) ( ) ( ) ( ) =, g P P P P P P = 2 2 l 2 exp 2 g σ π σ

30 Meter aalysis Fidig state sequece Classificatio 30 The most likely sequece of meter estimates ca be foud usig Viterbi algorithm causal algorithm: meter estimate at time is determied accordig to the ed-state of the best partial path at that time ocausal meter estimates after seeig a complete sequece of observatios ca be computed usig backward decodig

31 Classificatio 3 Phase estimatio (beat locatios) The model fids periods first ad the the phases for the differet levels of the meter phase estimatio is based o the last outputs of the resoator of the wiig period (i.e., the filter state) probabilistic modelig for phase is very similar to that for period estimatio, but is estimated separately for beat/measure/tatum

32 Meter aalysis Summary Classificatio 32 Registral accet sigals Degree of accetuatio (stress) as a fuctio of time at four frequecy chaels Metrical pulse stregths Meter Stregths of differet metrical pulses at time (resoator eergies) tatum, tactus, measure

33 Demostratios Classificatio 33

A. P. Klapuri, A. J. Eronen, J. T. Astola, Analysis of the meter of acoustic musical signals, IEEE Trans. Speech and Audio Proc. (in press).

A. P. Klapuri, A. J. Eronen, J. T. Astola, Analysis of the meter of acoustic musical signals, IEEE Trans. Speech and Audio Proc. (in press). A. P. Klapuri, A. J. Eroe, J. T. Astola, Aalysis of the meter of acoustic musical sigals, IEEE Tras. Speech ad Audio Proc. (i press). This material is preseted to esure timely dissemiatio of scholarly