Tow Mehnize Musi Pegogy Lingfeng Yng Stnfo Univesity stt We n pt existing tehniques fo the sttistil moeling of musi to the moeling of the qulity of musil pefomne, esulting in fine-gine sttistil unestning of hness in musi pefomne. This ives I-ssiste pplitions fo impoving skill. We pesent moeling esults using vious poilisti moels to moel pefomne in the simplifie setting of hythm gmes. 1 Intoution Lening musil instument is h. Iniviully, thousns of olls in fees e spent tking lessons, n yes spent ptiing t suoptiml level. Ptie without onstnt expet supevision often involves negleting the el tehnil shotomings n wose, lening the wong hits. The e stte in whih we e essing ou shotomings n lening the ight hits is known s eliete ptie [Eisson et l. 1993]. How n mhines help us e in this stte moe often y utomtilly pointing out wht nees wok though the sttistil moeling of plye pefomne? Fist is the eutionl vlue of hving the sttistis lone. Fo exmple, if we know we miss note sequene ptiully often though the sttistis, we eome ette infome of wht note sequenes to ptie. Seon is the utomtion of seony pegogil elements suh s ptie songs; y otining the most-likely-to-miss note sequenes, we n use tehniques fom lgoithmi musi to synthesize ptie song tht speifilly tgets ptiul tehnil wekness. Thi, is n ojetive, etile meti of impovement; if n ute sttistil moel fome fo peiting misse note sequenes eomes inesingly inute fo t the plye fees it in the futue, pesumly fte ptiing n getting ette, we n tie tehnil impovement to the lowee empiil poilities of misse note sequenes vesus the peite poility. In this ppe we pesent how the vile-oe Mkov moel, moel often use to sttistilly moel musi, my e use to fom similly etile moels of the qulity of musil pefomne. 2 Relte Wok This is n extension of the wok in [Yng 2010]. Musi ognition. The most elte fiel woul e in musi ognition. Musi ognition is onene out ette unestning how people elte to musi. Reently, thee hs een some inteest in moeling musi ognition [Honing 2006]. It hs not een estlishe yet wht goo moels e in this spe. Seeing how well ou popose moels peit plye pefomne my le to ette unestning of how h songs e h n esy ones, esy. Vieo gmes n plye moeling. The next most elte fiel is in vieo gmes. Reently, thee hve een emi eseh effots in moeling plyes of vieo gmes. [Peesen et l. 2009] isusses how to moel fuzzy onepts suh e-mil: lyng@s.stnfo.eu s fun n stisftion in pltfom gmes. Thee is lso wok in moeling plye pefomne, ut fo iffeent gme types. [Dhen et l. 2009]. lgoithmi musi. Finlly, the moels we pln on using n lso e foun in lgoithmi musi, whih is onene with the moeling n genetion of musi itself. [onklin 2003] povies suvey of sttistil tehniques. [Momk 1996] gives n oveview of musi genetion tehniques. Thee is lso wok in using sttistil moels to peit the nmes of songs [ohu n De Feits 2003]. In ptiul, we popose to pt suh methos fom moeling o geneting musi to peiting plye pefomne. oly, we e inteeste in gining sttistil, musi-theoeti unestning of skill evelopment in musil pefomne. Mkov moels. Mkov moels hve titionlly een use to otin etile sttistis of sequentil infomtion, inluing musi. The simplest kin of Mkov moel is the Mkov hin, whee eh meme of sequene of nom viles is onitionlly epenent on the pevious meme. Vile-oe Mkov moels (VMMs), the lss of moel use in this wok, e n extension of Mkov hins tht llow ontext-speifi onitioning on ontexts of vying length. We w ou methos hevily fom [egleite et l. 2004] whih gives n oveview of the ville tehinques fo tining VMMs. Othe litetue oveing VMM tining exist, suh s [Shulz et l. 2008]. eent evelopment is Mkov nom fiels, whih itionlly genelize the shpe of onitioning ontexts. In ft, Mkov nom fiels wee use eenty in [Lvenko n Pikens 2003] to moel polyphoni musi. 3 Polem Domin In hythm gme, the plye hits notes y pessing uttons on n input evie. Eh utton on the input evie oespons to note. These input evies ten to e moele on tul instuments, ut the nume of notes they enoe is muh lowe; ommonly, the nume of notes is oun 5. Duing the ouse of the gme, the plye s gol is hit inoming notes in time to the musi; i.e., fte stting gme, t eh point in sequene of peetemine time intevls, the plye hs the hne to hit speifi note within timing winow. t high levels of ply, this n eome vey iffiult; it is ommon fo hythm gme to eplite the notes in tehnilly iffiult el musil piee one fo one. We n fomlize these onepts s follows. Fist is the epesenttion of songs in the hythm gme: song is sequene of (note, time) pis: S := {s i = (n i, t i), n i P, t i R, t i t i+1} N i=1, whee N is the nume of notes in the song n eh n i is fom finite set of pithes P = {0, 1,..., H}. We my epesent the plye s input s simil sequene
I := {(n i, t i), n i N, t i t i+1} M i=1. Misse notes s skill meti. In this setting we efine skill s the ility to epoue the note sequene utely; i.e., the moe notes misse eltive to the sme song, the lowe the skill of the plye. Howeve, thee e still mny iffeent wys in whih to mesue whethe note is misse. One wy is timing mesuement; fo eh s i = (p i, t i) I suh tht thee is no s j = (n j, t j) S within timing winow of t 0 of t i; t i t j > t 0/2 j. In genel, this is oolen peite P S,I : S {0, 1}. Note tht this oes not ptue ll the wys of missing note. in ptiul, it oes not ptue the se whee the plye plys too mny notes. This then les to lele vesion of S, S I. S I := {(n i, t i, P (s i)), s i S, n i, t i R, t i t i+1} N i=1. Miss te. We n then onsie S I s ising fom some poility istiution P ove ll x i S I. We fomulte the miss te t eh note, the ontext-speifi poility tht the plye will miss note, se on this notion: Refeene Use input Lele efeene Timing winow time Figue 1: We tke efeene song (lue, top) n use input ove tht song (geen, mile) to otin leling of the efeene song with eh note s hit (lue) o misse (e) (ottom). Hee we give n exmple fo 3 pithes {,, }. We use 7 pithes in ou tul system. M i = P ((x i) 3 = 1 x 1... x i 1) sttistilly fine gine notion of miss te is impotnt euse it llows fo moe effetive pplitions fo skill evelopment. Note tht we o not yet el with polyphoni musi, whee (n i, t i) pis with the sme t i e llowe. uently eing investigte is pting the Mkov nom fiel metho fo olleting sttistis on polyphoni musi [Lvenko n Pikens 2003] to olleting sttistis on pefomne qulity tking polyphonis into ount. 4 System Oveview To seve s pltfom fo this eseh, we hve evelope hythm gme tht is spinoff of the etmni IIDX [Konmi 1999] hythm gme; it ugments the egul gme with t olletion mehnism. The gme is omptile with existing file fomts use y the etmni moing ommunity; with this, we my test ou ies on tlog of songs tht e ley ville. In typil session, the pplition my e plye like the oiginl etmni IIDX; the plye stts the gme n selets fom menu of songs to ply, plys the song, n n optionlly ply nothe song. Unlike most hythm gmes, howeve, etile t is ollete pe ply; fte plying eh song, the sequenes S lele with misse notes fom I n the timestmp of the song long with text elimiting the ply session is ppene to log file. In ition to the t olletion e fetues tht epesent explotions into how we n pply the esults of moeling the t to ssisting skill evelopment. We inlue fility fo geneting ptie songs tht onsist of epets of the most likely note sequenes tht esult in misse note. s the sope of this ppe els with the moeling spet, we will not go into it hee. Leling of t. hol n othe timing infomtion is then stippe to poue sting with lphet onsisting of onseutive hit/miss symols s i {Hn, Mn} whee n is one of P musil pithes: H M H H M H M H M... The miss te t note i my then equivlently e lulte s M i = P (s i = Mn pith(s i) = n, s 1... s i 1). 5 Moeling In the setting of mehnizing musi pegogy, thee e thee spets we equie of ny moel tht n elize the poility quey ove. 1. The moel mkes ute peitions. This is le; the quey given ove shoul e le to peit wht notes the plye will get wong. 2. The moel genelizes. The quey given ove shoul e le to pefom these peitions on t tht is not fom the tining set. 3. The moel is esilient to t stvtion. euse we tget iniviul plyes, one shoul not e foe to genete vst tset of pefomnes efoe the moel is usle. The quey shoul give ute esults fom single plye on single plythough of single song. n estlishe metho fo moeling poilities ove stings is the Mkov hin of oe k, lso known s the k-gm moel. Eh onseutive ouene of k symols in the sting is mthe ginst the symol tht omes next. y umulting list of (k-sequene, next symol) pis, one lso umultes the onitionl poility P (s i s i 1... s i k ). These moels n e vey ute. Howeve, in this se we e inteeste in the ontextul miss te fom the eginning of the entie song. If we wee to use fixe-oe Mkov moels, this woul equie multiple moels of vey high oe (fo typil song, in the hunes). This woul equie
n enomous mount of pefomne t fom eh plye; s this mens the moel is not esilient to t stvtion we nnot use k-gm moel fo ou puposes. 6 Vile-Oe Mkov Moels euse fixe-oe moels nnot e pplie ptilly to ou moeling polem, we tun to vile-oe Mkov moels (VMMs). VMMs e n extension of fixe-oe Mkov moels wheein one is llowe to onition on ny ontext of ity length, not just those of fixe length, s is the se in fixe-oe Mkov (n-gm) moels. gfeef gfeef gfeef gfeef Figue 2: possile VMM fo sting. llowing onitioning on ity-length ontexts lets one onentte the poility mss to ontexts of iffeent lengths, potentilly esulting in muh ette uy n geneliztion while equiing muh less t. 6.1 Tining the VMM Sevel lgoithms exist fo lening vile-oe Mkov moels. Essentilly wht nees to e one to len VMM is to pik ontext to mth to eh symol in the lphet, n then len the poilities using these (symol, ontext) pis. The vition is in how ontexts e selete. To gin high uy n geneliztion, goo VMM lening lgoithms genelly len ontexts tht mximize the likelihoo of the tining sequene x T 1 = x 1... X T : ˆP (x T 1 ) = T ˆP (x i x 1... x i 1). i=1 The likelihoo is tken to e the joint poility of ll symols in the sequene ouing given ll pevious symols. It is not tivil to etemine whih ontexts will o this given finite omputtionl esoues. Sevel lgoithms selet these ontexts se on infomtion-theoeti piniples; in ft, ny lossless ompession lgoithm tht is se on stoing itiony of phses itionlly inues VMM ove the input sequene [egleite et l. 2004]. The LZ-78 lgoithm. One suh ompession lgoithm is the LZ-78 lgoithm, whih ompesses well (heives high likelihoo) while lso eing extemely fst. The LZ-78 lgoithm woks s follows (see Figue 2): sting is onsume fom eginning to en, uiling up phse itiony. The shotest phse not in the itiony is pse n insete into the itiony t eh step. The LZ-MS lgoithm. The LZ-MS lgoithm is vint. It hs two pmetes, M n S. LZ-MS pefoms the LZ- 78 lgoithm on S 0 1-symol shifts of the input sting n ktks y M symols eh time phse is pse, umulting moe omplete itiony in the poess. Futhe etils e ville in [egleite et l. 2004]. We employ the LZ-MS vint in ou VMM lening lgoithm. Figue 3: The LZ-78 lgoithm pplie to the sting. ε Figue 4: The eision tee oesponing to the phse itiony of. Fom phse itiony to eision tee. The LZ-78 lgoithm poues set of phses given n input sting. Thee is oesponing tee fom whih onitionl queies of ny ontext length my esily e lulte. We my onsie the phse set s eing genete y tking noneteministi wlks in the tee up to one level efoe the leves. Futhe etils of how this is woks is ville in [egleite et l. 2004]. onsie the phses {,,,,,, }. The fist hte in eh of the phses must e fom the lphet {,,,, }, hene the fist level in the tee is {,,,, }. Next, we onsie the seon hte given eh of the fist htes; given, it n e {,, }, n given, thee e no hoies, n given, it n e {}. We poee in simil fshion until ll the phses in the itiony e exhuste, then nothe lphet {,,,, } to the tee. n exmple of onitionl poility lute with wlks in the tee: on(n) = (n)/ 7 Results s sis(n) {n} (s) P ( ) = on(ɛ ɛ ) = 5/33 test of the LZ-MS-se VMM s peition uy n geneliztion ws un on single plythough of single song. The esulting lele sting ompise 700 symols in length. The VMM ws tine on 600 symols with ontiguous setion of 100 symols left out to seve s the test set.
Reeive opeting hteisti (vg pefomne in leve-100-out oss vlition on single 700-note plythough 1.0 Tue positive te 0.8 0.6 Rnom pefomne 0.4 VMM_LZMS VMM_LZMS_ontext3 1-gm 2-gm 0.2 3-gm 4-gm 5-gm 0.0 0.0 0.2 0.4 0.6 Flse positive te 0.8 1.0 Figue 5: The eeive opeting hteisti of the LZ-MS VMM moel on the peition quey P (x i = Mn pith(x i) = n, x 1... x i 1). Note the infeio pefomne of oth the LZ-MS VMM moel n 1-to-5-gm moels on the simplifie fixe-oe-k peition queies P (x i = Mn pith(x i) = n, x i 1... x i k ). Reeive opeting hteisti. In Figue 5 we see tht the lgoithm hieves ette thn nom pefomne in its eeive opeting hteisti, whih is otine y using the peite poility ˆp long with some theshol poility θ [0, 1] whee if ˆp > θ, we eie tht the note ws misse, n if the note tully is some Mn, it is tue positive, else it is flse positive. The eeive opeting hteisti is the uve otine y plotting tue/flse positive pefomne ove sevel θ [0, 1]. The sme evlution meti is use in [Lvenko n Pikens 2003]. We elieve simil meti is ppopite fo peiting misse notes s it is fo note ouenes in genel. ompison with othe moels. Thee e two eisions we me in ou setting tht emn ompison: one is the hoie of VMMs ove fixe-oe Mkov moels. nothe is the hoie of onitioning stting fom the vey eginning of the song P (x i = Mn pith(x i) = n, x 1... x i 1) whih is vile-oe quey, vesus on some fixe-oe ontext P (x i = Mn pith(x i) = n, x i 1... x i k ). In Figue 5, we see tht peition uy n geneliztion ility of the LZ-MS VMM lgoithm is ipple when given the fixe-size quey, while fixe-oe moels o ette with the fixe-size quey, they o not o s well s the LZ-MS VMM lgoithm on the vile-oe quey. Finlly, the ility of the vile-oe quey to peit note misses is emonstte to e supeio to tht of the fixesize quey, t lest with these ptiul eliztions (VMMs n fixe-oe Mkov moels). We o not inlue fixe-oe moel esults on the moe vile-oe quey euse it equies the lening of n oe-t-lest-100 Mkov moel, whih in this setting woul lely not wok ue to t stvtion; unning suh moel on ou tining set esulte in unifom nswes, s the moel lene vey low poility of seeing ny note onitione on length-100 ontext. etinly othe ltentives exist, like smoothe omintions of fixe-oe moels n ltentive wys of lening VMMs. They e not inlue euse we onsie these essentilly VMMs, n ou ompison is etween VMMs n non-v MMs, not mong VMMs. Pefomne. The fste pefoming lening lgoithm is, the ette it n e use in el-time pplition suh s hythm gmes. It is ette to get feek in fom of n upte istiution quikly fte plying eh song. Fo the 700-note tining set efeene in Figue 5, the lening lgoithm onstute the phse itiony n eision tee on the oe of tenths of seons on n 2.66 GHz Intel Xeon (using 1 oe). Miss-te queies ppohe el-time pefomne (on the oe of huneths of seons). This pefomne pofile llows the lening lgoithm to e use fo evey song tht the plye plys, n the peition lgoithm fo iniviul notes in plythough. The lening n peition lgoithms wee implemente in Hskell with little eg to spe leks fom lzy evlution n miniml memoiztion; even highe pefomne shoul e in eh though pogm optimiztions. 8 Limittions n Futue Wok ugmenttions to the system. One ntul next step is to push the softwe out to moe uses n hve the ollete t e ville in lou-sevies-like fshion. This woul enle the olletion of t t lge sle, in tun enling the usge of lening lgoithms tht equie muh moe t ut n potentilly e muh moe esiptive. In ptiul, it woul enle plye to ompe pefomne vesus othe plyes, n to see how othe plyes impove thei tehnique, in tems of how the lene sttistis hnge fo nothe plye se on wht songs she plye. Mkov Rnom Fiels. esies not eing le to el with polyphoniity, the itei of VMMs tht ontext e ontiguous is quite limiting. Thee is potentil to use ette moel. We e uently evluting Mkov Rnom Fiels using the metho of [Lvenko n Pikens 2003]. The wk of this metho e tht it is muh slowe thn using VMMs with LZ-MS; it oul only e use in n offline mnne. Howeve, it my pove to e moe ute. The fetues lene y the metho my lso e qulittively insightful to the musiin inteeste in impoving tehnique. Intetive mhine lening. potentilly useful ugmenttion to ny ontext-se poility moel, whih eltes to intetive mhine lening, is to let the ontexts e seletle y the use, fixing set of them, n then seleting the est se on some othe lgoithm. This is euse often the use hs petty goo ie of wht kins of musil ontexts they mke mistkes in. Othe pplitions. uently onsiee pplitions inlue the utomti synthesis of h enough ptie songs n tools to tk tehnil pogess t fine-gine level. Wht othe pplitions e thee? Finlly, the genel e of I-ssiste humn lening is pomising eseh ietion. We ve spent lot of time tehing omputes how to o things ette so muh tht the omputes might s well e tehing us.
Refeenes egleite, R., El-Yniv, R., n Yon, G. 2004. On peition using vile oe Mkov moels. Jounl of tifiil Intelligene Reseh 22, 1, 385 421. ohu, E., n De Feits, N. 2003. Nme Tht Song! Poilisti ppoh to Queying on Musi n Text. vnes in neul infomtion poessing systems, 1529 1536. onklin, D. 2003. Musi genetion fom sttistil moels. In Poeeings of the IS 2003 Symposium on tifiil Intelligene n etivity in the ts n Sienes, itesee, 30 35. Dhen,., noss,., n Ynnkkis, G. 2009. Plye moeling using self-ogniztion in tom ie: unewol. In Poeeings of the IEEE Symposium on omputtionl Intelligene n Gmes (IG2009), Milno, Itly. http://www. itu. k/ ynnkkis/ig09 IOI. pf. Eisson, K., Kmpe, R., n Tesh-R ome,. 1993. The ole of eliete ptie in the quisition of expet pefomne. PSYHOLOGIL REVIEW-NEW YORK- 100, 363 363. Honing, H. 2006. omputtionl moeling of musi ognition: se stuy on moel seletion. Musi Peeption 23, 5, 365 376. Konmi. 1999. etmni IIDX. Lvenko, V., n Pikens, J. 2003. Polyphoni musi moeling with nom fiels. In Poeeings of the eleventh M intentionl onfeene on Multimei, M, 120 129. Momk, J. 1996. Gmm se musi omposition. omplex systems 96, 321 336. Peesen,., Togelius, J., n Ynnkkis, G. 2009. Moeling plye expeiene in supe mio os. In IEEE Symposium on omputtionl Intelligene n Gmes, 2009. IG 2009, 132 139. Shulz, M., Weese, D., Rush, T., D oing,., Reinet, K., n Vingon, M. 2008. Fst n ptive vile oe mkov hin onstution. lgoithms in ioinfomtis, 306 317. Yng, L. 2010. Moeling Plye Pefomne in Rhythm Gmes. To ppe in: SIGGRPH si 2010 tehnil skethes.