Sound Transformations Based on the SMS High Level Attributes

Transcription

1 Sound Trnsformtons Bsed on the SMS Hgh Level Attrbutes Xver Serr, Jord Bond Audovsul nsttute, Pompeu Fbr Unversty mbl 3, 82 Brcelon, Spn {serr, Abstrct The bsc Spectrl Modelng Synthess (SMS) technque models sounds s the sum of snusods plus resdul. Though ths nlyss/synthess system hs proved to be successful n trnsformng sounds, more powerful nd ntutve muscl trnsformtons cn be cheved by movng nto the SMS hgh-level ttrbute plne. n ths pper we descrbe how to etrct hgh level sound ttrbutes from the bsc representton, modfy them, nd dd them bck before the synthess stge. n ths process new problems come up for whch we propose some ntl solutons. ntroducton The gol of the reserch behnd SMS hs lwys been to get sound representtons bsed on nlyss tht re musclly ntutve nd from whch sound trnsformtons cn be obtned wthout genertng ny rtfcts n the syntheszed sound. Through ths work t hs become cler tht t s mpossble to obtn sngle representton sutble for every sound nd pplcton. t hs requred to etend the orgnl determnstc plus stochstc model [] n order to nclude specfc dervtons for prtculr stutons, whle mntnng the overll nlyss/synthess frmework s generl s possble [2]. The most fleble representton cn be cheved from the non rel tme nlyss of well recorded sounds n dry envronment. These sounds should be pseudo-hrmonc nd monophonc, tht s, melodes or sngle notes, plyed wth muscl nstrument. As these condtons re reled, one by one, other representtons re possble tht re not s fleble but tht re powerful enough for mny musc nd udo pplctons. n ths rtcle we concentrte on the most fleble representton possble, thus gvng up some of the generlty of SMS. However, t stll shres the sme nlyss/synthess frmework wth ll other representtons bsed on SMS, under whch, representtons cn esly be found tht re optml for every sound nd pplcton. We wll strt by descrbng the concept of hgh level ttrbutes. We dscuss the fct tht n ll nturl sounds most of these ttrbutes re nterrelted, thus mkng the nlyss hrder. We lso hve to mntn these nterreltons, or t lest py ttenton to them, when trnsformtons re ppled. Then, we descrbe the etrcton of these ttrbutes from the bsc snusodl plus resdul representton nd dscuss the ssue of trnsformtons bsed on ths new representton. 2 Hgh level ttrbutes The ccomplshment of menngful prmeterzton for sound trnsformton pplctons s dffcult tsk. We wnt prmeterzton tht offers n ntutve control over the sound trnsformton process, wth whch we cn ccess most of perceptul ttrbutes of sound, or even group of relted sounds, such s ll the sounds produced by gven nstrument. The bsc SMS nlyss results n smple prmeterzton pproprte for descrbng the mcrostructure of sound. These prmeters re the nstntneous frequency, mpltude nd phse of ech prtl nd the nstntneous spectrl chrcterstcs of the resdul sgnl. There re other useful nstntneous ttrbutes tht gve hgher level bstrcton of the sound chrcterstcs. For emple we cn descrbe fundmentl frequency, mpltude nd spectrl shpe of snusodl component, mpltude nd spectrl shpe of resdul component, nd totl mpltude. These ttrbutes re esly clculted t ech nlyss frme from the output of the bsc SMS nlyss. At the sme level, there re other ttrbutes useful for the chrcterzton of other spects of the sound mcrostructure, lke the degree of hrmoncty, nosness, spectrl tlt, nd spectrl centrod. A prt from the nstntneous, or frme, vlues, t s lso useful to hve prmeters tht chrcterze the tme evoluton of the sound. These tme chnges cn

2 be descrbed by the dervtves of ech one of the nstntneous ttrbutes. Another mportnt step towrds musclly useful prmeterzton s the segmentton of sound nto regons tht re homogeneous n terms of ts sound ttrbutes. Then we cn dentfy nd etrct regon ttrbutes tht wll gve hgher level control over the sound. The smplest, nd most generl, segmentton process dvdes melody nto notes nd slences nd then ech note nto n ttck, stedy stte nd relese regons. Globl ttrbutes tht cn chrcterze ttcks nd releses refer to the verge vrton of ech of the nstntneous ttrbutes, such s verge fundmentl frequency vrton, verge mpltude vrton, or verge spectrl shpe chnge. n the stedy stte regons t s menngful to etrct the verge of ech of the nstntneous ttrbutes nd mesure other globl ttrbutes such s tme-vryng rte nd depth of vbrto. The concept of sound ttrbutes cn be tken step further by consderng the common ttrbute vlues of n entre nstrument,.e., for ll the sounds produced by the nstrument. The ttrbutes of ech note re compred nd combned n order to group the chrcterstcs tht re common to the whole nstrument, or some of ts sounds, levng ech note only wth the dfferences from the verge vlues of the dfferent ttrbutes. Ths gves muscl control t the nstrument level wthout hvng to ccess ech ndvdul nlyzed note. 3 Attrbute correltons n most cses the dfferent ttrbutes of gven sound re correlted nd chnge n one s ccompned by chnges n others. Some of these correltons re due to the coustc behvor of the ctul vbrtng system producng the sound, others to the wy tht the system s ected by the plyer. All muscl nstruments ehbt ths property nd we hve to py ttenton s to the wy we clculte, etrct nd put bck these ttrbutes, n order to preserve the chrcter of gven nstrument nd specfc performnce of t. For muscl resons we mght wnt to brek these correltons n the syntheszed sound, thus chngng the nturl behvor of the nstrument nd the perceptul reltons between these ttrbutes. But, even n ths cse, the results wll be more nterestng f we understnd these correltons nd defne rules tht generte non-nturl correltons between the dfferent perceptul ttrbutes of sound. Some of the correltons re well known by muscns, nd sound desgners tke them nto ccount when they wnt to emulte the behvor of n coustc nstrument usng gven synthess technque. Emples of relevnt correltons found n nstrumentl sounds re between fundmentl frequency nd spectrl shpe, between fundmentl frequency nd mpltude, or between mpltude nd spectrl shpe. For emple, n most nstruments, s we go up the scle (hgher fundmentl frequency) the number of prtls decreses, the spectrl slope ncreses nd, generlly, the mpltude lso ncreses. Also, s we ply louder the resultng sound becomes brghter (fltter spectrl slope). n the contet of SMS there re correltons tht come up ether n the nlyss or the synthess of sound. Emples of these re between mpltude of snusods nd mpltude of resdul nd between mpltude of prtls, frequency of prtls nd spectrl shpe of sound. f we wnt to mke trnsformtons to sound whle mntnng ts chrcter we hve to ccompny the chnge of n ttrbute wth the pproprte chnges to the correlted ttrbutes. 4 Attrbute detecton nd etrcton From the bsc snusodl plus resdul representton t s qute smple to etrct the ttrbutes mentoned bove. The crtcl ssue s how to etrct them n order to mnmze nterferences, thus obtnng, s much s possble, menngful hgh level ttrbutes free of correltons. We frst etrct nstntneous ttrbutes nd ther dervtves, then we segment the sound, nd fnlly we cn etrct regon ttrbutes. 4. Frme ttrbutes The bsc frme, or nstntneous, ttrbutes re: mpltude of snusodl nd resdul component, totl mpltude, fundmentl frequency, spectrl shpe of snusodl nd resdul component, hrmonc dstorton, nosness, spectrl centrod, nd spectrl tlt. These ttrbutes re obtned t ech frme usng the nformton tht results from the bsc SMS nlyss nd not tkng nto ccount the dt from prevous or future frmes. Some of them cn be etrcted from the frme dt, levng normlzed frme, others re nformton ttrbutes tht descrbe the chrcterstcs of the frme nd re not etrcted from the orgnl dt. The mpltude of the snusodl component s the sum of the mpltudes of ll hrmoncs of the current frme epressed n db, AS totl = 2 log = where s the lner mpltude of the th hrmonc nd s the totl number of hrmoncs found n the current frme.

3 The mpltude of the resdul component s the sum of the bsolute vlues of the resdul of the current frme epressed n db. Ths mpltude cn lso be computed by ddng the frequency smples of the correspondng mgntude spectrum, A totl = 2 log = 2 log M N X () k where s the resdul sound, M s the sze of the frme, X (k) s the spectrum of the resdul sound, nd N s the sze of the mgntude spectrum. The totl mpltude of the sound t the current frme s the sum of ts bsolute vlues epressed n db. t cn lso be computed by summng the mpltudes of the snusodl nd resdul components, A totl = 2 log = 2 log M = = 2 log X () k + N N X () k where (n) s the orgnl sound nd X (k) spectrum. s ts The fundmentl frequency s the frequency tht best eplns the hrmoncs of the current frme. Ths cn be computed by tkng the weghted verge of ll the normlzed hrmonc frequences, F = = f = where f s the frequency of the th hrmonc. A more complete dscusson on the ssue of fundmentl frequency n the contet of SMS cn be found n [3]. The spectrl shpe of the snusodl component s the envelope descrbed by the mpltudes nd frequences of the hrmoncs, or ts ppromton, {( f )( f, )...( )} Sshpe =, 2 2 f, The spectrl shpe of the resdul component s n ppromton of the mgntude spectrum of the resdul sound t the current frme. A smple functon s computed s the lne segment ppromton of the spectrum, { e, e2,, eq, en M } = m[ X ( qm + k) ] k = M 2, M 2 +,,, M 2, nd shpe =, where k M s the number of frequency smples used for ech clculton of locl mmum. Other spectrl ppromton technques cn be consdered dependng on the type of resdul nd the pplcton. The hrmonc dstorton s mesure of the degree of devton from perfect hrmonc prtls, HrmDstorton = = f ( F ) = The nosness s mesure of the mount of non snusodl nformton present n the frme. t s computed by tkng the rto of resdul mpltude versus totl mpltude, Nosness = n M = M elted nosness mesures result from studyng the shpe of ech spectrl pek nd ts devton from the del snusodl pek. The spectrl centrod s the mdpont of the energy dstrbuton of the mgntude spectrum of the current frme. One mght lso thnk of t s the blnce pont of the spectrum, Centrod = N k N f s X N () k X ( k) The spectrl tlt of the snusodl component s the slope of the lner regresson of the dt ponts f,, ( ) Stlt = 2 = t = t f 2 σ = where t = f, nd σ s weght σ 2 σ = fctor for ech dt pont tht we hve found useful to set t proportonl to the mpltude vlue, = σ σ =.

4 For completeness we could lso compute the spectrl tlt of the resdul component but we hve not consdered relevnt ttrbute for our purposes. 4.2 Frme vrton ttrbutes The frme to frme vrton of ech ttrbute s useful mesure of ts tme evoluton, thus n ndcton of chnges n the sound. t s computed n the sme wy for ech ttrbute, Vl ( l) Vl( l ) = M where Vl(l) s the ttrbute vlue for the current frme nd Vl( l ) s the ttrbute vlue for the prevous one. 4.3 Segmentton Sound segmentton hs proved mportnt n utomtc speech recognton nd musc trnscrpton lgorthms. For our purposes t s very vluble s wy to pply regon dependent trnsformtons. For emple, tme stretchng lgorthm would be ble to trnsform the stedy stte regons, levng the rest unmodfed. The smplest, nd most generl, segmentton process dvdes melody nto notes nd slences nd then ech note nto n ttck, stedy stte nd relese regons. Attck nd relese regons re dentfed by the wy the nstntneous ttrbutes chnge n tme nd the stedy stte regons re detected by the stblty of these sme ttrbutes. The technques orgnlly developed for speech [4], bsed on Pttern-ecognton, Knowledge-Bsed or Neurl Network methodologes, strt to be used n musc segmentton pplctons [5]. Most of the pproches pply clssfcton methods tht strt from sounds fetures, such s the ones descrbed n ths pper, nd re ble to group sequences of frmes nto predefned ctegores. No relble nd generl purpose technque hs been found. Our eperence s tht they requre nrrowng the problem to specfc type of muscl sgnl or ncludng user nterventon stge to help drect the segmentton process. 4.4 egon ttrbutes Once gven sound hs been segmented nto regons we cn study nd etrct the ttrbutes tht descrbe ech one. Most of the nterestng ttrbutes re smply the men nd vrnce of ech of the frme ttrbutes for the whole regon. For emple, we cn compute the men nd vrnce for the mpltude of snusodl nd resdul components, the fundmentl frequency, the spectrl shpe of snusodl nd resdul components, or the spectrl tlt. Vbrto s specfc ttrbute present n mny stedy stte regons of sustned nstrumentl sounds tht requres specl tretment. There s nother rtcle from our reserch group tht descrbes ths ssue n detl [6]. egon ttrbutes cn be etrcted from the frme ttrbutes n the sme wy tht the frme ttrbutes were etrcted from the low level SMS dt. The result of the etrcton of the frme nd regon ttrbutes s herrchcl mult-level dt structure where ech level represents dfferent sound bstrcton. 5 Attrbute trnsformtons The herrchcl dt structure tht ncludes complete descrpton of gven sound offers mny possbltes for sound trnsformtons. Most musclly menngful trnsformtons re done by modfyng severl ttrbutes t the sme tme nd t dfferent bstrcton levels. Hgher level trnsformtons cn refer to spects lke sound chrcter, rtculton or epressve phrsng. These des led to the development of front ends such s grphcl nterfces [7] or knowledge-bsed systems [8] tht re ble to del wth the complety of ths sound representton. 6 Concluson n ths rtcle we hve presented n overvew of the work beng crred out t the Audovsul nsttute n the drecton of etendng the SMS sound representton towrds hgher level bstrctons. t opens new reserch problems tht wll led to nterestng nd ectng musc pplctons. eferences [] X. Serr. Muscl Sound Modelng wth Snusods plus Nose. G. D. Pol nd others (eds.), Muscl Sgnl Processng, Swets & Zetlnger Publshers, 997. [2] X. Serr nd others. ntegrtng Complementry Spectrl Models n the Desgn of Muscl Syntheszer. Proceedngs of the CMC, 997. [3] P. Cno. Fundmentl Frequency Estmton n the SMS Anlyss. Proceedngs of the

5 Dgtl Audo Effects Workshop (DAFX98), 998. [4] E. Vdl nd A. Mrzl. A evew nd New Approches for Automtc Segmentton of Speech Sgnls. L. Torres nd others (eds.), Sgnl Processng V: Theores nd Applctons, Elsever Scence Publshers, 99. [5] S. ossgnol nd others. Feture Etrcton nd Temporl Segmentton of Acoustc Sgnls. Proceedngs of the CMC, 998. [6] P. Herrer. Vbrto Etrcton nd Prmeterzton n the Spectrl Modelng Synthess Frmework. Proceedngs of the Dgtl Audo Effects Workshop (DAFX98), 998. [7] A. Loscos nd E. esn. SmsPerformer: A el-tme Synthess nterfce for SMS. Proceedngs of the Dgtl Audo Effects Workshop (DAFX98), 998. [8] J. L. Arcos nd others. Se: Cse-Bsed esonng System for Genertng Epressve Muscl Performnces. Proceedngs of the CMC, 997.