ISE 599 Cmputatial Mdelig f Expressive Perfrmace Playig Mzart by Aalgy: Learig Multi-level Timig ad Dyamics Strategies by Gerhard Widmer ad Asmir Tbudic Preseted by Tsug-Ha (Rbert) Chiag April 5, 2006 Authr Gerhard Widmer Machie Learig ad Patter Recgiti Kwledge Discvery i Databases / Data Miig / Text Miig Itelliget Music ad Audi Prcessig Cmputatial Mdels f Expressive Music Perfrmace Music Ifrmati Retrieval (MIR) Authr Itrducti Asmir Tbudic The gal is t lear t apply sesible temp ad dyamics shapes at varius levels f the hierarchical musical phrase structure. Mrever, the paper ivestigated t what extet a machie ca autmatically build peratial mdels f certai aspects f perfrmace via iductive learig frm real perfrmaces by highly skilled musicias. The paper preseted a tw-level apprach t learig bth phrase-level ad te-level timig ad dyamics strategies fr expressive music perfrmace. Widmer & Tbudic: Playig Mzart by Aalgy
ISE 599 Cmputatial Mdelig f Expressive Perfrmace April 5, 2006 Steps 1. Decmpsig give expressi curves it elemetary patters that ca be assciated with idividual phrases at differet phrase levels, i rder t btai meaigful traiig examples fr phrase-level learig, ad t separate phrase-level effects frm lcal te-level effects. 2. The earest eighbr algrithm predicts phrase-level expressive shapes i ew pieces by aalgy t shapes idetified i similar phrases i ther pieces. 3. The rule learig algrithm PLCG lears predicti rules fr te-level effects frm the residuals that cat be attributed t the phrase structure by the expressi decmpsiti algrithm. 4. Cmbiig phrase-level ad te-level predictis Multilevel decmpsiti f expressi curves The scres f musical pieces + measuremets f lcal temp ad dyamics variatis Bth temp ad ludess are represeted as multiplicative factrs, relative t the average temp ad dyamics f the piece. The hierarchical phrase structure f the pieces is aalyzed by had. Usig the class f secd-degree plymials t apprximate the expressive curves. Multilevel decmpsiti f dyamics curve f perfrmace f Mzart Sata K.279:1:1, mm.31 38 Cmputig the plymial that best fits the part f the curve that crrespds t this phrase, ad subtract the temp r dyamics deviatis explaied by the apprximatis. Subtract = Divide After the iteratis, the fial curve left is called residual curve. Origial dyamics curve plus the secd-rder plymial givig the best fit at the tp phrase level (blue). Widmer & Tbudic: Playig Mzart by Aalgy
ISE 599 Cmputatial Mdelig f Expressive Perfrmace April 5, 2006 Each shw, fr successively lwer phrase levels, the dyamics curve after subtracti f the previus apprximati, ad the best-fittig apprximatis at this phrase level. Recstructi (red) f the rigial curve by the fur levels f plymial apprximatis. Phrase-level learig via earest eighbr predicti Residual after all higher-level shapes have bee subtracted. Give a phrase i a ew piece, the algrithm searches its memry fr the mst similar phrase i the kw pieces (at the same phrase level) ad predicts the plymial assciated with this phrase as the apprpriate shape fr the ew phrase. A bvius drawback f earest eighbr algrithms is that they d t prduce explicit, iterpretable mdels they make predictis, but they d t describe the data ad the target classes. Widmer & Tbudic: Playig Mzart by Aalgy
ISE 599 Cmputatial Mdelig f Expressive Perfrmace April 5, 2006 Rule-based learig (PLCG) f residuals Cmbiig phrase-level ad te-level predictis The residuals ca be expected t represet a mixture f ise ad meaigful r iteded lcal deviatis. PLCG is a iductive rule learig algrithm that has bee shw t be highly effective i discverig reliable, rbust rules frm cmplex data where ly a part f the data ca actually be explaied, ad it lears sets f classificati rules fr discrete classificati prblems. It starts with a iitial flat expressi curve (i.e., a list f 1.0 values) ad the successively multiplies the curret value by the phrase-level predictis ad the te-level predicti. Fr a give te i that is ctaied i m hierarchically ested phrases pj, j = 1... m, the expressi (temp r dyamics) value expr(i) t be applied t it is cmputed as m expr( ) = pred ( ) f ( set ( )) i PLCG i p j j= 1 where pred PLCG ( i ) is the te-level predicti f temp r dyamics made by the residual rules leared by PLCG f pj is the apprximati plymial predicted as beig best suited fr the j th -level phrase p j by the earest-eighbr learig algrithm. p j i Experimets ad quatitative results The mea squared errr ( pred( i ) - expr( i )) MSE = Â The mea abslute errr pred( i ) - expr( i ) MAE = Â The crrelati betwee predicted ad real curve. If the relative errr MSEL r MAEL, MSED MAED L=the perfrmace prduced by the learer D=the default (mechaical, iexpressive) perfrmace is less tha 1.0, that meas the curves predicted by the learer are clser t the piaist s actual perfrmace tha a purely mechaical rediti. i= 1 i= 1 2 Results, by sata sectis, f crss-validati experimet. Widmer & Tbudic: Playig Mzart by Aalgy
ISE 599 Cmputatial Mdelig f Expressive Perfrmace April 5, 2006 Musical results: A case f success Quadratic r parablic apprximatis might t be as suitable fr describig expressive timig. The dyamics curves are geerally better apprximated by the plymials tha the temp curves. The te-level rules d ideed imprve the quality f the results, bth i terms f errr ad crrelati. The imprvemet may be slight i quatitative terms, but listeig tests shw that the predicted residuals ctribute imprtat audible effects that imprve the musical quality f the resultig perfrmaces. Learer s predictis fr the dyamics curve f Mzart Sata K.280, 1st mvemet, mm. 25 50. Quadratic expressi shapes predicted fr phrases at fur levels (blue) Cmpsite predicted dyamics curve resultig frm phrase-level shapes ad te-level predictis (red) vs. piaist s actual dyamics (black). Oly temp ad dyamics were shaped by the system. Articulati ad pedalig are igred. Grace tes ad ther ramets are curretly iserted via a simple way. Numeric errr ad musical quality d t always crrelate. PLCG ideed discvered quite geeral ad sesible priciples f lcal timig ad dyamics. Widmer & Tbudic: Playig Mzart by Aalgy
ISE 599 Cmputatial Mdelig f Expressive Perfrmace April 5, 2006 Musical results: A case f failure Learer s predictis fr the temp curve f Mzart Sata K.332, 2d mvt., mm.1 8.5. The shapes predicted fr the varius phrases (phrase structure idicated by brackets belw) learer s predicted temp curve (grey, withut markers) vs. piaist s actual curve (black, with markers). Recstructi f piaist s temp f Mzart K.332:2, mm.1 8, by hierarchy f fur phrase shapes. The shapes assciated with the varius phrases (phrase structure idicated by brackets belw) Recstructed temp curve (grey, withut markers) vs. piaist s actual curve (black, with markers). Widmer & Tbudic: Playig Mzart by Aalgy
ISE 599 Cmputatial Mdelig f Expressive Perfrmace April 5, 2006 Discussi & Limitatis This iterpretati prduced by the learig algrithm suds terrible, especially with respect t timig. Actually it des t have may mistakes the part f the learer, but e icrrectly chse shape ca cmpletely destry the musical acceptability f a passage. The temp curve cat be well apprximated by phrase-level shapes. There are may lcal timig deviatis i the piaist s perfrmace that are essetial t the musical effect cat be captured by the phrase-level apprximatis The phrase structure aalysis might have bee perfrmed at t glbal a level. The prpsitial attribute-value represetati whm the paper used des t allw the leaer t refer t details f the iteral structure ad ctet f phrases. Nearest eighbr learig is that it des t prduce iterpretable mdels. It is t simplistic that predictig phrasal shapes idividually ad idepedetly f the shapes assciated with ther related phrases. Widmer & Tbudic: Playig Mzart by Aalgy