SOUND SEPARATION OF POLYPHONIC MUSIC USING INSTRUMENT PRINTS

Transcription

1 SOUND SEPARATION OF POLYPHONI MUSI USING INSTRUMENT PRINTS Krstó Azél 1 and Szabols Ivánsy 2 Department o Automaton and Appled Inormats Budapest Unversty o Tehnology and Eonoms 3-9. Muegyetem rkp. H-1111 Budapest Hungary azelkr@aut.bme.hu 1 vansy@aut.bme.hu 2 ABSTRAT Deomposng a polyphon musal reordng to separate nstrument traks or notes has always been a hallenge. Suh a sgnal s the superposton o many separate traks and t s theoretally mpossble to extrat the omponent traks wthout the normaton that was lost at the superposton. Ths paper ntrodues a new way o sound separaton o mono-aural dgtal reordngs. The proposed algorthm nputs the lost normaton by usng a model o real nstruments n order to make the separaton o ndvdual musal notes possble. The separaton method manly targets the proessng and orreton o musal reordngs that annot be re-reorded. 1. INTRODUTION Our long term nterest n sound separaton s motvated by the problem o orretng exstng musal reordngs adjustng volumes o nstruments separately xng msplayed notes et. Although muh work has been presented on audo soure separaton restorng ndvdual notes that are present n the reordng s stll unsolved. However the human audtory system s very eetve n derentatng between sound soures and ndvdual notes. We an blok out unwanted nose voes o other speakers n a rowded envronment we an ous on ertan nstruments n a polyphon musal pee. Even today we know very lttle about how the human bran exatly works. However the at that we are able to magne the sound o derent nstruments even n omplete slene or that we an reognze the voe o our relatves wthout seeng ther ae learly shows that we store memores o propertes o derent sounds. Ths leads us to the assumpton that the human bran uses ths a pror normaton or real-tme separaton o the mus we hear. Ths assumpton s onrmed n stuatons when we hear new unusual nstruments. Untl we get to know the eatures o the new sound soure (whh may only take a ew seonds or mnutes) our separaton apablty s rather lmted wthout ths a pror normaton. Ths paper proposes a separaton algorthm that s apable o separatng ndvdual notes even n mono reordngs. The mportane o requrng only one hannel les n the at that the soluton s sutable also or older reordngs or reordngs whh may have been reorded to two or more hannels but the orgnal traks are or some reason no more avalable. I the soure reordng has more than one hannel then other tehnques are also avalable n addton to the soluton desrbed n ths paper to get even better results. 2. RELATED WORK [1] s a sound soure separaton algorthm that requres no pror knowledge and perorms the task o separaton d purely on azmuth dsrmnaton wthn the stereo eld. The results are mpressve. However separatng ndvdual notes s not n the ous. [4][5][6] desrbe a method whh separates harmon sounds by applyng lnear models or the overtone seres o the sound. The method s d on a two-stage approah: ater applyng a multpth estmator to nd the ntal sound parameters more aurate snusodal parameters are estmated n an teratve proedure. Separatng the spetra o onurrent musal sounds s d on the spetral smoothness prnple [3]. Beamormng tehnques [10] along wth the Independent omponent Analyss ramework oer a derent way o separaton. However these methods rely on ertan prelmnary ondtons and studo setup to aheve good results. In [7][8] derent transormaton methods were studed n order to determne the best possble means or analyss and proessng o reorded dgtalzed polyphon mus sgnals. Ths paper deals wth an approah that ams to separate sngle notes rom the remanng part o the reordng. The ous s on the qualty o the output sgnals rather than the speed or automaton level o the proess. 3. ONEPT Fgure 1 shows the blok dagram o the algorthm we are gong to dsuss n detal. Frst the sgnal s transormed nto requeny-doman usng FFT transorm. Ater the transormaton speal algorthms are appled to the resultng spetrogram n order to retreve the detals and prese normaton that annot be extrated dretly rom the FFT results. Seton 4 deals wth the detals o these algorthms ousng manly on Frequeny Estmaton (FE) and Phase Memory (PM). Seton 5 ntrodues the nstrument model that s used n our envronment to support the separaton proess EURASIP 931

2 Seton 6 goes through the steps o the atual separaton. Ater the separaton the separated sound hannels (the solated notes and the remander o the orgnal reordng) are transerred bak to tme doman. Fnally Seton 7 summarzes the results and the perormane o the system draws the onlusons and ponts out mprovement possbltes. REORDING FFT + FREQUENY ESTIMATION FREQ RESTORATION + INVERSE FFT REORDING REMAINDER INSTRUMENT SAMPLES FFT + FREQUENY ESTIMATION STORED MODELS SEPARATION VELOITY AND VOLUME ESTIMATOR MAGNITUDE SPLITTER MUSIAL SORE (USER INPUT) FREQ RESTORATION + INVERSE FFT ISOLATED NOTES Fgure 1 - Blok dagram o sound separaton 4. PREISE SPETROGRAM ALULATION Fast Fourer Transorm s used or onvertng the sound data rom tme doman to requeny doman. We use overlappng sgnal ragments (rames) to analyze the sound sgnal. In ontrast to Fourer Transorm whh operates n ontnuous requeny spae FFT deomposes the sgnal to a sum o dsreet requeny values. As t s known ths auses smearng n the spetrogram whh makes t hard to get the exat requenes out o the orgnal sound sgnal. However at polyphon sound sgnal analyss t s neessary to mnmze the eet o the smearng and get the prese requenes o snusodal omponents. In order to overome ths problem several steps are taken. Frst wndowng s appled to eah rame n tme-doman [7]. Ths wll redue the smearng eet o the FFT to some extent. The resultng sgnal s then proessed wth FFT. The resoluton o the resultng mage n requeny doman s stll not satsatory. Ths paper ntrodues the Frequeny Estmaton (FE) method whh s an extenson to [11]. FFT desrbes the sgnal n terms o snusods that have a well dened bn requeny phase and magntude. Any snusodal soure sgnal wth a requeny that mathes one o the bn requenes wll produe magntude only on one spe bn whle other requenes wll produe magntudes on several neghbour bns leavng no lue on the prese requeny that was orgnally present n the sgnal. Frequeny Estmaton method nds the requeny or eah bn by analysng the phase normaton on the same bn n suessve rames. The orgnal method ntrodued n [11] ompares two suessve rames. The requeny o a bn s alulated as ollows. k ( S/ K) k expt expt + ( t t ) 2Π + l 2Π l Z k + 2 Π ( t t ) 2 1 k Π< +Π where S s the sample rate o the sgnal; K s the rame sze; k and kt represent respetvely the bn requeny and exp t phase o bn k n tme t; s the expeted phase; k t kt s the ane between the expeted and measured phase; s the estmated requeny o bn k n tme t. The k t greater the tme derene between the start o rames the more prese the estmated value o k t. On the other hand bg tme derenes lmt the maxmum detetable dstane between and k. k t To overome ths lmtaton and urther mprove the preseness o the alulaton an extenson to the orgnal algorthm s proposed. The requenes an be ound more presely by takng the weghtened average o the last m phase atons ( s the oeent o bn k n tme t). Ths extenson wll be reerred to as Phase Memory (PM). ˆ x tx kt t t x m tx t t x m ˆ k + 2 Π ( t t ) 2 1 Fgure 2 shows the eet o the Frequeny Estmaton and Phase Memory on a spetrogram. (Spetrograms are plotted n two dmensons (requeny and tme) wth graysale olors ndatng the magntude). Fgure a) plots the raw spetrogram; b) the spetrogram wth [11] appled and ) shows the eet o the Phase Memory method. I the reordng s not very omplex the spetrogram n ) s understandable even to human eyes EURASIP 932

3 vertal uzzy area at the begnnng that s aused by the pano hammer represents the aperod omponent. Fgure 3 - waveorm o a pano note (tme doman) deomposed to aperod (hammer) and perod (strngs) parts Fgure 2 - spetrogram: a) FFT b) FFT+FE wthout PM ) FFT+FE 5. INSTRUMENT SAMPLES To understand the eatures o a sound reordng we must dsuss the eatures o separate musal nstruments rst. In general the sound o a musal nstrument n any gven short moment n tme an be deomposed nto two man omponents: a perod and an aperod sound omponent. Perod sounds are those emtted by a soure that produes regular vbratons over tme resultng n a olleton o requenes alled harmons partals or overtones. Harmon requenes orgnatng rom the same soure are related n a way that they our n multples o the lowest requeny reerred to as the undamental requeny. Thus a olleton o harmonally related requenes o whh the undamental s 200 Hz would our wth requenes o 400 Hz 600 Hz 800 Hz 1000 Hz 1200 Hz et. Aperod sounds are those whh most oten our pereptually as nose (ymbal rash drums snare or the sound o the pano hammer at the begnnng o eah pano note see Fgure 3). Aoustally nose s dened as a random olleton o requenes rom a sngle soure whh are not harmonally related and whose waveorm s thereore rregular. Most nstruments generate both harmonally related requenes and nose-lke transents. I we want to elmnate a note rom the reordng we must rst examne one sngle note o that nstrument wth the same undamental requeny veloty and ampltude. We must keep n mnd that veloty and ampltude are not synonyms here. A key on the pano an be pressed harder and soter (veloty derene) resultng n derent spetrograms even normalzed beore omparng; whle the same key-press an be reorded wth derent mrophone gan settngs (volume derene) the spetrograms o whh ater normalzng wll resemble eah other [9]. Fgure 4 plots a pano note at 260Hz n requeny doman. The spetrogram shows ts two man omponents. Horzontal lnes represent the perod omponent whh slowly deays n tme n ase o a pano note. The short Human hearng s lmted to about Hz dependng manly on age. Our model stores the magntudes n the spetrogram o notes o the nstrument n ths range. Sne t s pratally mpossble to store all possble notes an nstrument s able to generate only a ew are stored at derent requenes and velotes. The number o needed samples s subjet o uture researh the urrent mplementaton works wth 3-5 veloty levels per nstrument and 6-8 sampled requenes per otave. I the spetrogram o an unsampled note s needed later n the separaton algorthm the mssng sample s nterpolated rom exstng ones. I enough samples are stored the derene n the output qualty wll not be noteable. Fndng the requred number o sampled notes s out o the sope o ths paper. From now on ths model wll be reerred to as nstrument prnt or smply prnt. Fgure 4 - Plot o a pano note at 260Hz The model an shortly desrbed as ollows. I we take the spetrogram oeents ( k ) o a note on a ertan requeny wth veloty M then A _ M rt r 05 r R ˆ 2 R < kt < r R 24 logr 2 ˆ kt 2007 EURASIP 933

4 where A represents one nstrument sample startng at t0. A M denotes the sum o the energy o a narrow st requeny band n tme t; r denotes the dstane rom the requeny and R s an expermental value whh denes the sze o the requeny band. 6. SEPARATION Ater we have the nstrument prnts and an produe the rght prnt or any requeny and veloty by nterpolaton we an move on to the soure reordng to be proessed. The spetrogram o the rght prnt wll be separated rom the remanng part o the reordng usng lnear deomposton. Assumng that we know the exat requeny volume and veloty o a ertan note that we want to separate rom the remanng part o the reordng the ollowng algorthm an be proposed or the separaton. The phase and magntude normaton o the spetrograms wll be handled separately. The phases o the resultng spetrograms wll be or all notes () the same as the orgnal phases whle the magntudes o the reordng wll be splt between them. org reman org + org reman 1 The mplementaton o the atual note separaton dvdes the energy between the target notes teratvely n D steps: α [0]0 t [0] S 0 [0] t M st Tstart [ d] rt per [ d]0 r 05 r ˆ < 12 < 2 [ d] t [ d] [ d] A [ ] αrt δ [ d] 1 (1 ) D [ d] k S S + ( ) [ d+ 1] t [ d] t [ d] 1 t [ d] t... [ d+ 1]0 t [ d] It... [ D]0 t [ D 1] It I 2 r 05 R < ˆ ˆ < 2 otherwse r + 05 R the separaton [ D] s the remanng energy n the reordng ater the separaton whle M reers to the volume and veloty values whh are assumed to be known. Ater the separaton the spetrograms an be transormed bak to tme doman. In the above paragraphs the startng tme requeny volume and veloty o the note to be separated were assumed to be known. These startng parameters are needed by the separaton algorthm. However ths does not resemble a real-le senaro at all. Generally when gven a good representaton o the sound sgnal (spetrogram musal sore et.) the user an nteratvely nput the requeny and tme o the note on a termnal qute presely. On the other hand speyng the veloty and volume s usually a muh harder task sne the average user does not ether reognze or understand the derene between these two expressons. Thereore t s sae to eed the startng tme and requeny as parameters to the separaton engne but the veloty and volume must be alulated algorthmally. For the above mentoned problem we propose an teraton algorthm that s d on the gradent method. The desred volume-veloty pars are approxmated wth ntal values and ater a number o repettons the optmal values an be approahed. The algorthm s as ollows: 1. The user nteratvely nputs all the onurrent notes exstng n the musal seton to be proessed. We requre normaton on all note startng and endng tmes and requenes. However no normaton s requred on volume or veloty values ths tme. Ths s a reasonable ompromse between onvenene or the user and omplexty n the algorthm. 2. An ntal volume and veloty level s determned or all notes. (From now on a (start end requeny volume veloty) ouple wll be reerred to as a 'regon'). Ths ntal level an saely set at 100% the strength o our stored prnts. We ound that the ntal volume level has no nluene on the outome o the algorthm; however t may aet the overall speed. 3. The separaton algorthm s run usng the seleted volume and veloty values. 4. The error o the separaton s alulated: Err rem Tend K [ D] t Tstart k 0 pr M rt [ D] t T r 05 start r 2 R ˆ < kt r+ 05 ˆ 2 R kt < Err α Err + (1 α) Err Tend Err A s sum rem pr where T start s the attak tme o note D s the number o steps [d] s the urrent step S s the spetrum o note ater 2007 EURASIP 934

5 where Err rem sums the error aused by remanng energy on the bns n the reordng ater the separaton Err pr sums the error there was less energy n the reordng than the prnts requred Err sum s the global error o the separaton step wth the urrent regon parameters T start and T end are respetvely the startng and endng tmes o the observed tme. r s the regon denter n the observed tme sle and 0<a<1 s the qualty preerene parameter o the separaton. Hgh a value means preerene on the qualty o the separated notes to the qualty o the remanng part whle low a value provdes better remander qualty but poorer separated note qualty. 5. Volume level o one o the regons s slghtly nreased (e.g. rom 100% to 101%). Error s ounted agan. Ths proedure s repeated wth all the regons n the observed tme sle. 6. Same as 5 wth veloty levels. 7. The gradent vetor an be omputed rom the error values. Ths vetor shows the dreton n whh we should hange the urrent volume and veloty values to redue the error values. Volume and veloty values are moded aordngly. 8. Steps 3-7 are repeated as many tmes as neessary to get prese enough volume and veloty values. 9. Ater ndng the desred reord level and veloty values gvng the lowest possble error value the separaton algorthm s run wth the alulated parameters. The applaton o the gradent method s only possble there s only one loal mnmum n the error-spae otherwse t ould lead the algorthm towards the wrong dreton. Provng that the volume-veloty level spae meets ths ondton s out o the sope o ths paper. We must menton that there are notes n the orgnal reordng that are loated losely n requeny t ntrodues the beatng eet. Ths eet s not resolved by the urrent algorthm. Ths ssue s the subjet o uture researh. 7. SUMMARY The paper showed a method or separatng sngle nstrument notes rom a reordng usng pre-reorded nstrument prnts. The results are qute promsng. An example reordng and ts separated notes an be downloaded rom However experments are needed or a mathematal valdaton. For reordngs that only ontan harmonally unrelated notes the algorthm provdes very lear results and even some notes are loated on eah other s or overtone requenes the separaton provdes reasonably good results. However n these ases mprovement s stll requred to deal wth beatng and get hgher qualty output. The other area o uture researh s buldng more lexble nstrument models. We annot always have prnts or all the possble notes o an nstrument n most ases we do not even have aess to the orgnal nstrument the reordng was taken wth. Thus as we do not use the rght prnt only a lose approxmate we may experene some dstorton n the separaton output whh mght be audble also to less audophle lsteners. REFERENES [1] Barry D. Lawlor R. and oyle E. Sound Soure Separaton: Azmuth Dsrmnaton and Resynthess n Pro. 7th Internatonal onerene on Dgtal Audo Eets DAFX 04 Naples Italy [2] Samer A. Abdallah and Mark D. Plumbley Polyphon transrpton by non-negatve sparse odng o power spetra n Pro ISMIR 2004 Barelona Span Otober [3] Klapur A. Multpth estmaton and sound soure separaton by the spetral smoothness prnple n Pro. IEEE Internatonal onerene on Aousts Speeh and Sgnal Proessng Salt Lake ty USA [4] Vrtanen T. and Klapur A. Separaton o Harmon Sounds Usng Multpth Analyss and Iteratve Parameter Estmaton n Pro. IEEE Workshop on Applatons o Sgnal Proessng to Audo and Aousts New York USA [5] Vrtanen T. and Klapur A. Separaton o Harmon Sound Soures Usng Snusodal Modelng n Pro. IEEE Internatonal onerene on Aousts Speeh and Sgnal Proessng Istanbul Turkey [6] T. Vrtanen and A. Klapur Separaton o harmon sounds usng lnear models or the overtone seres n Pro. IEEE Internatonal onerene on Aousts Speeh and Sgnal Proessng (IASSP '02) Orlando Fla USA May [7] K. Azél Sz. Ivánsy Musal soure analyss wth DFT n Pro. MroAD 2006 Internatonal Sent onerene Mskol Hungary Marh [8] K. Azél Sz. Ivánsy Musal soure analyss: spetrogram versus ohleagram n Press MroAD 2007 Internatonal Sent onerene Unversty o Mskol (Mskol Hungary) Marh [9] K Azél Manpulaton o Musal Reordngs Usng Instrument Prnts n Pro. o Automaton and Appled omputer Sene Workshop 2006 (AAS'06) Budapest Hungary June 2006 [10] N. Mtanouds M. E. Daves Usng Beamormng n the audo soure separaton problem 7th Int Symp on Sgnal Proessng and ts Applatons Pars July 2003 [11] tt.html ( ) 2007 EURASIP 935