MODELING SPEECH PARAMETER SEQUENCES WITH LATENT TRAJECTORY HIDDEN MARKOV MODEL. Hirokazu Kameoka

Transcription

1 15 IEEE INTERNATIONAL WORKSHOP ON MACHINE LEARNING FOR SIGNAL PROCESSING, SEPT. 17, 15, BOSTON, USA MODELING SPEECH PARAMETER SEQUENCES WITH LATENT TRAJECTORY HIDDEN MARKOV MODEL Hirokazu Kameoka Nippon Telegraph and Telephone Corporaion / The Univeriy of Tokyo ABSTRACT Thi paper propoe a probabiliic generaive model of a equence of vecor called he laen rajecory hidden Markov model (HMM). While a convenional HMM i only capable of decribing piecewie aionary equence of daa vecor, he propoed model i capable of decribing coninuouly imevarying equence of daa vecor, governed by dicree hidden ae. Thi feaure i noeworhy in ha i can be ued o model many kind of ime erie daa ha are coninuou in naure uch a peech pecra. Given a equence of oberved daa, he opimal ae equence can be decoded uing he expecaion-maximizaion (EM) algorihm. Given a e of raining example, he underlying model parameer can be rained by eiher he expecaion-maximizaion algorihm or he variaional inference algorihm. Index Term Sequenial modeling, Hidden Markov model (HMM), Trajecory HMM, Laen rajecory HMM, Expecaion-Maximizaion algorihm, variaional inference 1. INTRODUCTION The weakne of hidden Markov model (HMM) i ha hey have difficuly in modeling and capuring he local dynamic of feaure equence due o he piecewie aionariy aumpion and he condiional independence aumpion on feaure equence. Tradiionally, in peech recogniion yem, hi limiaion ha been circumvened by appending dynamic (dela and dela-dela) componen o he feaure vecor. HMM-baed peech ynhei yem [1] alo ue he join vecor of aic and dynamic feaure a an oberved vecor in he raining proce. In he ynhei proce, on he oher hand, a equence of aic feaure i generaed according o he oupu probabiliie of he rained HMM given an inpu enence by aking accoun of he explici conrain beween he aic and dynamic feaure []. Alhough he HMM-baed peech ynhei framework ha many aracive feaure, one drawback i ha he crieria ued for raining and ynhei are inconien. While he join likelihood of aic and dynamic feaure i maximized during he raining proce, he likelihood of only he aic feaure i maximized during he ynhei proce. Thi implie ha he model parameer are no rained in uch a way ha he generaed pa- Thi work wa uppored by JSPS KAKENHI Gran Number 6731 and 686. The auhor would like o hank Dr. Tomoki Toda (NAIST) for fruiful dicuion and Mr. Tomohiko Nakamura (Univeriy of Tokyo) for hi help wih conducing he experimen. rameer equence become opimal. To addre hi problem, Zen [3] inroduced a varian of HMM called he rajecory HMM, which wa obained by incorporaing he explici relaionhip beween aic and dynamic feaure ino he radiional HMM. Thi ha made i poible o provide a unified framework for he raining and ynhei of peech parameer equence, however, i caue difficuly a regard parameer inference. Since he condiional independence aumpion on he feaure vecor i lo, efficien algorihm for raining and decoding regular HMM uch a he Vierbi algorihm and he Forward-Backward algorihm are no longer applicable o he rajecory HMM. Thu, ome approximaion and brue-force mehod are uually neceary o obain raining and decoding algorihm [3, 4]. In hi paper, we propoe formulaing a new model called he laen rajecory HMM. In conra wih he convenional rajecory HMM, he preen model pli he generaive proce of an oberved feaure equence ino wo procee, one for a equence of he join vecor of aic and dynamic feaure given HMM ae and he oher for an oberved feaure equence given he equence of he join vecor. By reaing he join vecor of aic and dynamic feaure a a laen variable o be marginalized ou, we obain a probabiliy deniy funcion of an oberved feaure equence wih a differen form from he likelihood funcion of he rajecory HMM. A decribed below, hi new formulaion naurally allow he combined ue of powerful inference echnique uch a he expecaion-maximizaion (EM) algorihm, Vierbi algorihm and Forward-Backward algorihm for raining and decoding, while ill reaining he piri of he original rajecory HMM. Thi work i no only direced oward peech ynhei applicaion bu alo oward everal differen applicaion uch a voice converion and acouic-o-ariculaory mapping, in which rajecory modeling ha proven o be effecive [5 7]. Anoher inereing applicaion we have in mind i audio ource eparaion. Recenly, we propoed mehod for ingle- and muli-channel audio ource eparaion baed on facorial HMM [8 11], he pecrogram of a mixure ignal i modeled a he um of he oupu emied from muliple HMM, each repreening he pecrogram of an underlying ource. One promiing way o improve hi approach would be o incorporae he dynamic of ource pecra. Thi can be accomplihed by plugging he preen model ino he facorial HMM formulaion. The preen formulaion will play a key role in making hi poible /15/$31. c 15 IEEE

2 . TRAJECTORY HIDDEN MARKOV MODEL We ar by briefly reviewing he original formulaion of he rajecory HMM [3]. Le u ue c o denoe a D- dimenional aic feaure vecor and define he join vecor of c and i velociy and acceleraion componen o := [c T, c T, c T ] T R 3D a he oberved vecor a ime. We wrie he equence of he aic feaure and he oberved vecor a c = [c T 1,..., c T T ]T and o = [o T 1,..., o T T ]T, repecively. Thu, he dimenion of c and o become DT and 3DT. The relaionhip beween c and o can be decribed explicily uing a conan 3DT by DT marix W a o = W c, (1) W i a pare marix ha append fir and econd order ime derivaive o he aic feaure vecor equence. Wihin he radiional HMM framework, a equence of oberved vecor, o, i imply aumed o be generaed from an HMM. Here, if we aume he emiion probabiliy deniy o be a ingle Gauian diribuion, he probabiliy deniy funcion of o given a ae equence = [ 1,..., T ] and an HMM parameer e λ = {µ, U, π}, wih µ = {µ i } 1 i I, U = {U i } 1 i I, and π = {π i,j } 1 i I,1 j J, i given a p(o, λ) = N (o; µ, U ) = T N (o ; µ, U ), () =1 N (x; µ, Σ) denoe a Gauian diribuion wih mean µ and covariance Σ: N (x; µ, Σ) 1 e 1 Σ 1 (x µ)t Σ 1 (x µ). (3) µ and U denoe he mean equence and a block diagonal marix whoe diagonal elemen are given by he equence of he covariance marice of he emiion deniie over ime: µ = [µ T 1,..., µ T T ] T, (4) U = diag(u 1,..., U T ). (5) In HMM-baed peech ynhei yem, he parameer e i ypically rained by olving he maximum likelihood eimaion problem ˆλ = argmax λ log p(o, λ)p(), (6) p() i given by he produc of he ae raniion probabiliie. A he ynhei age, given a ae equence and wih he rained parameer λ, a aic feaure equence c i generaed according o p(c, λ) i defined a ĉ = argmax p(c, λ), (7) c p(c, λ) N (W c; µ, U ) (8) e 1 (ct W T U 1 W c ct W T U 1 µ ) (9) Fig. 1. Illuraion ha how he difference beween HMM and rajecory HMM. = N (c; c, V ). (1) By compleing he quare in he exponen of (9) wih repec o c, we immediaely obain c and V a c = (W T U 1 W ) 1 W T U 1 µ, (11) V = (W T U 1 W ) 1. (1) Thu, he oluion o (7) i c. Geomerically, (1) can be viewed a a cuing plane of he deniy p(o, λ) a o = W c. A hown above, he radiional HMM-baed framework ue differen crieria for raining and ynhei: While p(o, λ) i ued for raining, p(c, λ) i ued for ynhei. Thi implie ha ˆλ i no necearily opimal for generaing opimal c. To addre hi inconiency beween he raining and ynhei crieria, Zen [3] propoed inroducing a framework called he rajecory HMM, which alo ue (1) a he raining crierion. Inead of olving (6), he parameer e λ i hu rained by olving {ˆλ, ŝ} = argmax λ, log p(c, λ)p(), (13) c i reaed a he oberved daa. Unlike he regular HMM, he condiional independence aumpion on oberved vecor doe no hold in he rajecory HMM: While he regular HMM aume ha each oberved vecor depend only on he curren ae, (1) indicae ha he oberved vecor c a each frame depend on he enire ae equence. Thi implie ha i i difficul o direcly apply he efficien decoding and raining algorihm ued in he HMM framework (uch a he Vierbi algorihm and he Forward-Backward algorihm). Thu, ome approximaion and brue-force mehod are uually neceary o perform raining and decoding [3]. Becaue of hi, he decoding algorihm i no guaraneed o find he opimal ae equence and he raining algorihm i no guaraneed o converge o a local opimal oluion. Noe ha hi alo applie o he minimum generaion error (MGE) raining framework [4], which ue (1) in which V i replaced by an ideniy marix a he raining crierion.

3 3.1. Model 3. LATENT TRAJECTORY HMM While he HMM i only capable of decribing piecewie aionary equence of daa vecor, he rajecory HMM i capable of decribing coninuouly varying equence of daa vecor, governed by dicree hidden ae. Thi feaure i noable in ha i can be ued o model many kind of ime erie daa ha are coninuou in naure, however, i caue a difficuly a regard parameer inference. We propoe inroducing a concepually imilar framework baed on a differen formulaion, which i advanageou in ha i alleviae he difficuly relaed o parameer inference. Inead of reaing o a a funcion of c, we rea o a a laen variable ha i relaed o c hrough a of conrain o W c. The relaionhip o W c can be expreed hrough he condiional diribuion p(c o). For example, we can define p(c o) a { p(c o) exp 1 } (W c o)t Λ(W c o), (14) Λ i a conan poiive definie marix ha can be e arbirarily. Indeed, hi probabiliy deniy funcion become larger a o approache W c. By compleing he quare in he exponen of (14) wih repec o c, we can wrie p(c o) explicily a p(c o) = N (c; m c o, Λ 1 c o ), (15) m c o = Ho, (16) H = (W T ΛW ) 1 W T Λ, (17) Λ c o = W T ΛW. (18) By uing hi and p(o, λ) defined in (), we can wrie p(c, λ) a p(c, λ) = p(c o)p(o, λ)do, (19) in a differen way from (1). Geomerically, hi can be viewed a a marginal diribuion of he e of he projeced value of o ono he ubpace o = W c. From () and (15), he join likelihood p(c, o, λ) can be wrien a p(c, o, λ) = p(c o)p(o, λ) { ( [ ] ) T ( [ ] )} exp 1 c c m x Λ x m x, () o o [ ] m x = Λ 1 x U 1, (1) µ [ ] Λ Λ x = c o W T Λ ΛW H T Λ c o H + U 1. () Thu, p(x, λ) = N (x; m x, Λ 1 x ) x = [c T, o T ] T. By uing he blockwie marix inverion formula, Λ 1 x i given a [ ] Σ x = Λ 1 Σ cc Σ co x =, (3) Hence, (19) can be wrien a Σ oc Σ oo Σ cc = Λ 1 c o + HU H T, (4) Σ co = HU, (5) Σ oc = U H T, (6) Σ oo = U. (7) p(c, λ) = N (c; Hµ, Σ cc ). (8) We call hi model he laen rajecory HMM. Wih hi framework, given a ae equence and a parameer e λ, c i generaed according o ĉ = argmax p(c, λ). (9) c Obviouly, he oluion o hi i Hµ. I i imporan o noe ha wih hi framework, he parameer inference problem can be deal wih uing he Expecaion-Maximizaion (EM) algorihm by reaing he join vecor [c T, o T ] T a he complee daa. 3.. Decoding and raining algorihm A wih he rajecory HMM framework, he preen framework ue p(c, λ) for feaure equence generaion, ae decoding and parameer raining in a conien manner. The problem of ae decoding and parameer raining can be formulaed a he following opimizaion problem: ŝ = argmax {ˆλ, ŝ} = argmax λ, log p(c, λ)p() (3) log p(c, λ)p(). (31) Since he decoding problem (3) i a ubproblem of he raining problem (31), here we only derive an algorihm for olving he raining problem (31). By regarding he e coniing of c and o a he complee daa, hi problem can be viewed a an incomplee daa problem, which can be deal wih uing he Expecaion- Maximizaion (EM) algorihm. The likelihood of and λ given he complee daa i given by (). By aking he condiional expecaion of log p(c, o, λ) wih repec o o given c, = and λ = λ, and hen adding log p(q), we obain an auxiliary funcion Q(, λ) := E o c,,λ [log p(c, o, λ)] + log p(). (3) By leaving only he erm ha depend on and λ, Q(, λ) can be wrien a Q(, λ),λ = E o c,,λ [log p(o, λ)] + log p()

4 = 1 { log U + Tr ( U 1 R ) µ T U 1 ō + µt U 1 } µ + log p(), (33) ō = E o c,,λ [o] = µ + Σ ocσ 1 cc (c Hµ ), (34) R = E o c,,λ [oot ] = Σ oo Σ ocσ 1 cc Σ co + ōō T. (35) Here, he prime mark indicae he value obained uing he model parameer updaed a he previou ieraion. Since U i a block diagonal marix, a given in (5), (33) can be decompoed ino he um of T individual erm: Q(, λ),λ = 1 T =1 {log U + Tr[U 1 R ] µ T U 1 ō + µ T U 1 µ } T + log π 1 + log π 1,, (36) ō T = ō 1 ō =., R = R 1... R T. (37) Wih fixed λ, Q(, λ) can be maximized wih repec o by employing he Vierbi algorihm. Wih fixed, Q(, λ) i maximized wih repec o λ when 1[ = i]ō µ i = U i = π i,j =, (38) 1[ = i] 1[ = i](r ō µ T i µ i ō T + µ i µ T i ) 1[ = i] 1[ 1 = i, = j] 1[ 1 = i], (39), (4) if 1[ = i] i and j denoe ae indice and 1[ ] denoe an indicaor funcion ha ake he value 1 if i argumen i rue and oherwie. Overall, he parameer raining algorihm can be ummarized a follow: (E-ep) Subiue and λ ino and λ and recompue ō and R uing (34) and (35). (M-ep) Updae λ uing (38) (4) and find uing he Vierbi algorihm. = argmax Q(, λ) (41) Noe ha if λ i fixed, he above algorihm reduce o a ae decoding algorihm. I may appear ha a huge amoun of compuaion for invering Σ cc i required o compue ō and R. However, hi can be carried ou very efficienly. Fir, by uing he Woodbury marix ideniy, Σ 1 cc can be wrien a Λ c o Λ c o ((HU H T ) 1 +Λ c o ) 1 Λ c o. Nex, ince Λ can be e arbirarily, we e Λ a U 1 o compue (HU H T ) 1. Under hi eing, (HU H T ) 1 i given a W T U 1 W. Since boh W T U 1 W and Λ c o are pare ymmeric band marice, (W T U 1 W + Λ c o ) 1 Λ c o can be compued efficienly uing he Choleky decompoiion. To iniialize, one reaonable way would be o earch for = argmax p(o, λ) uing he Vierbi algorihm Variaional learning algorihm We decribe a differen approach for parameer raining baed on variaional inference. The random variable of inere in our model are o, and λ = {µ, P, π} µ = {µ i } 1 i I, P = {P i := U 1 i } 1 i I, and π = {π i,j } 1 i I,1 j J. We denoe he enire e of he above parameer a Θ = {o,, λ}. Our goal i o compue he poerior p(θ c) = p(c, Θ). (4) p(c) By uing he condiional diribuion defined in 3.1, we can wrie he join diribuion p(c, Θ) a p(c, Θ) = p(c o)p(o, λ)p(). (43) To obain he exac poerior p(θ c), we mu compue p(c), which involve many inracable inegral. Inead of obaining he exac poerior, we conider approximaing hi poerior variaionally by olving an opimizaion problem: argmin KL(q(Θ) p(θ c)), (44) q KL( ) denoe he Kullback-Leibler (KL) divergence beween i wo argumen, i.e., KL(q(Θ) p(θ c)) = q(o,, λ) q(o,, λ) log dodλ. (45) p(o,, λ c) By rericing he cla of he approximae diribuion o hoe ha facorize ino independen facor: q(o,, λ) = q(o)q()q(µ, P )q(π), (46)

5 we can ue a imple coordinae acen algorihm o find a local opimum of (44). I can be hown uing he calculu of variaion ha he opimal diribuion for each of he facor can be expreed a: ˆq(X) exp E Θ\X [log p(c, Θ)], (47) X indicae one of he facor and E Θ\X [log p(c, Θ)] i he expecaion of he join probabiliy of he daa and laen variable, aken over all variable excep X. From (47), he variaional diribuion are given in he following form: ˆq(o) = N (o; m, Γ), (48) ˆq(µ, P ) = N (µ i ; ρ i, (β i P i ) 1 )W(P i ; B i, ν i ), i (49) ˆq(π i ) = Dir(π i ; α i ), (5) he parameer are updaed via he following equaion m = m 1. m T R 1 r, (51) Γ 1 Γ =... Γ T i q( 1 =i)ν i B i O R =... O i q( T =i)ν i B i i q( 1 =i)ν i B i ρ i r =. i q( T =i)ν i B i ρ i R 1, (5) + HT Λ c o H, (53) + HT Λ c o c, (54) β i q( =i), (55) ρ i 1 q( =i)m, β i (56) B 1 i q( =i)(γ + m m T ) β i ρ i ρ T i, (57) ν i β i + 3D, (58) α i,j 1 + q( 1 =i, =j). (59) W and Dir denoe he Wihar diribuion and he Dirichle diribuion, repecively, defined a W(X; V, ν) X ν d 1 e 1 Tr(V 1 X), (6) Dir(x; α) i x αi 1 i, (61) X i a d d ymmeric marix of random variable ha i poiive definie and V i a d d poiive definie marix. q( = i) and q( 1 = i, = j) can be compued uing he forward-backward algorihm a a ubrouine, a in [1]. 4. EXPERIMENTS To confirm he generalizaion abiliy of he preen model and he convergence peed of he preen raining algorihm, we conduced parameer raining experimen uing mel-ceprum equence of peech a experimenal daa. We choe he parameer raining algorihm for he original rajecory HMM developed by Zen e al. a he baeline mehod. Zen algorihm ue he mehod of eepe acen for updaing λ and he delayed deciion Vierbi algorihm for updaing. Reader are referred o [3] for he deail. Since he degree of freedom of he rajecory HMM and he laen rajecory HMM are exacly he ame when he number of hidden ae are he ame, he difference of he log-likelihood core obained wih he preen and baeline algorihm would reflec he difference in heir generalizaion abiliie. The experimenal condiion were a follow. We ued 5 peech daa excerped from he ATR peech daabae, from each of which we obained he mel-ceprum equence of he fir 5 frame. For boh he propoed model and he rajecory HMM, he number of hidden ae were e a 14. Λ were fixed a Λ = diag(a,..., A), (6) }{{} T.1 1 A =.1 1. (63).1 1 The workaion ued o perform he experimen had an Inel Core i3-1 Proceor wih a 3.3GHz 4 clock peed and a 7.7GB memory. Fig. how he evoluion of he log-likelihood wih repec o he number of ieraion and compuaion ime during he parameer raining of he propoed model and he convenional rajecory HMM. A Fig. how, he preen algorihm converged faer han he convenional algorihm. Thi reveal he effecivene of he combined ue of efficien aiical inference echnique uch a he EM algorihm and he dynamic programming principle by he propoed algorihm. I i alo worh noing ha he converged value of he log-likelihood obained wih he propoed algorihm wa greaer han ha obained wih he convenional algorihm. Thi implie he poibiliy ha, compared wih he convenional model, he propoed model ha a higher generalizaion abiliy, namely an abiliy o fi an arbirary e of feaure equence, given ha he degree-of-freedom of he wo model were he ame. Fig. 3 how an example of parameer generaion uing he propoed model. Afer raining λ uing 5 peech daa, a feaure equence ĉ wa generaed according o (9) given a ae equence. The figure a he op how he pecrogram of a peech ample and he figure a he boom how he pecrogram conruced uing ĉ obained uing he rained λ and he ae equence labeled from he peech ample. A Fig. 3 how, he propoed model wa able o repreen he coninuouly ime-varying naure of peech pecrogram reaonably well, howing ha i ha a imilar propery o he rajecory HMM.

6 Log-likelihood Frequency (khz) # ieraion Propoed Trajecory HMM.5 3 x Time () Log-likelihood Propoed Trajecory HMM Time (min) Frequency (khz) 6 4 Fig.. Evoluion of he log-likelihood wih repec o he number of ieraion (op) and he compuaion ime (boom). 5. CONCLUSIONS Inpired by he rajecory HMM framework propoed by Zen e al., hi paper propoed a probabiliic generaive model for decribing coninuouly ime-varying equence of daa vecor governed by dicree hidden ae. The propoed model i advanageou over he convenional rajecory HMM in ha i make i poible o derive convergence-guaraneed and efficien algorihm for parameer raining and ae decoding. Inereing fuure work involve incorporaing he propoed model ino he facorial HMM formulaion o develop a new mehod for audio ource eparaion ha ake accoun of he dynamic of ource pecra. 6. REFERENCES [1] T. Yohimura, K. Tokuda, T. Mauko, T. Kobayahi, and T. Kiamura, Simulaneou modeling of pecrum, pich and duraion in HMMbaed peech ynhei, in Proc. The 6h European Conference on Speech Communicaion and Technology (EUROSPEECH 1999), 1999, pp [] K. Tokuda, T. Yohimura, T. Mauko, T. Kobayahi, and T. Kiamura, Speech parameer generaion algorihm for HMM-baed peech ynhei, in Proc. IEEE Inernaional Conference on Acouic, Speech and Signal Proceing (ICASSP ),, pp [3] H. Zen, K. Tokuda, and T. Kiamura, Reformulaing he HMM a a rajeory model by impoing explici relaionhip beween aic and dynamic feaure vecor equence, Compuer Speech and Language, vol. 1, pp , 7. [4] Y.-J. Wu and R.H. Wang, Minimum generaion error raining for HMM-baed peech ynhei, in Proc. 6 IEEE Inernaional Conference on Acouic, Speech and Signal Proceing (ICASSP 6), 6, pp [5] T. Toda, A.W. Black, and K. Tokuda, Voice converion baed on maximum-likelihood eimaion of pecral parameer rajecory, IEEE Time () Fig. 3. Example of parameer generaion uing he propoed model. The pecrogram of a raining ample (op) and he pecrogram conruced uing he generaed parameer (boom). Traancion on Audio, Speech, and Language Proceing, vol. 15, no. 8, pp. 35, 7. [6] T. Toda, A.W. Black, and K. Tokuda, Saiical mapping beween ariculaory movemen and acouic pecrum uing a Gauian mixure model, Speech Communicaion, vol. 5, no. 3, pp. 15 7, 7. [7] L. Zhang and S. Renal, Acouic-ariculaory modelling wih he rajecory HMM, IEEE Signal Proceing Leer, vol. 15, pp , 8. [8] M. Nakano, J. Le Roux, H. Kameoka, T. Nakamura, N. Ono, and S. Sagayama, Bayeian nonparameric pecrogram modeling baed on infinie facorial infinie hidden Markov model, in Proc. 11 IEEE Workhop on Applicaion of Signal Proceing o Audio and Acouic (WASPAA11), 11, pp [9] T. Higuchi, H. Takeda, T. Nakamura, and H. Kameoka, A unified approach for underdeermined blind ignal eparaion and ource aciviy deecion by mulichannel facorial hidden markov model, in Proc. The 15h Annual Conference of he Inernaional Speech Communicaion Aociaion (Inerpeech 14), 14, pp [1] T. Higuchi and H. Kameoka, Join audio ource eparaion and dereverberaion baed on mulichannel facorial hidden markov model, in Proc. The 4h IEEE Inernaional Workhop on Machine Learning for Signal Proceing (MLSP 14), 14. [11] T. Higuchi and H. Kameoka, Unified approach for underdeermined BSS, VAD, dereverberaion and DOA eimaion wih mulichannel facorial HMM, in Proc. of The nd IEEE Global Conference on Signal and Informaion Proceing (GlobalSIP 14), 14. [1] M.J. Beal, Variaional Algorihm for Approximae Bayeian Inference, Ph.D. hei, Gaby Compuaional Neurocience Uni, Univeriy College London, 1998.