Multi-Modal User Interaction Fall 2008

Transcription

1 Mul-Modal User Ineracon Fall 2008 Lecure 2: Speech recognon I Zheng-Hua an Deparmen of Elecronc Sysems Aalborg Unversy Denmark z@es.aau.dk Mul-Modal User Ineracon II Zheng-Hua an 2008 ar I: Inroducon Inroducon hsory and rends Speech sgnal represenaon emplae based approach DW Sascal model based approach HMM Varably Mul-Modal User Ineracon II Zheng-Hua an

2 ASR hsory. Specral resonances exraced by an analogue fler bank and logc crcus; phoneme syllable dg recognon 2. DW IWR IBM LV ASR Bell Labs SI ASR CMU CSR ARA 5- year proec 3. Sascal framework HMM CMU IBM J.Baker Δcepsrum Furu -gram IBM eural ne DARA program Resource managemen ask Dscrmnave approach robus ASR DARA program broadcas news EARS rch ranscrpon GALE sponaneous speech audovsual ASR S. Furu 2007 Mul-Modal User Ineracon II Zheng-Hua an I echnology progress Mul-Modal User Ineracon II Zheng-Hua an

3 97-976: he ARA proec ARA launched 5 year Spoken Undersandng Research proec Goal: 000 word vocabulary a few speakers connuous speech consraned grammar 90% undersandng rae near real me on a 00 MIS machne 4 Sysems bul by he end of he program SDC 24% BB s HWIM 44% CMU s Hearsay II 74% CMU s HARY 95% -- bu 80 mes real me! Ra Reddy HARY based on engneerng approach: search on nework of all he possble uerances Conclude: Speech Undersandng s oo early for s me Lesson learned: Hand-bul knowledge does no scale up eed of a global opmzaon creron Mul-Modal User Ineracon II Zheng-Hua an s -- he Sascal Approach Hdden Markov Models based sascal approach Fred Jelnek and Jm Baker IBM Foundaons of modern speech recognon engnes Wˆ arg max A W W W Acousc HMMs a a22 Word r-grams a33 w w w 2 a2 a23 S S2 S3 J Baker o Daa Lke More Daa Whenever I fre a lngus our sysem performance mproves 988 Some of my bes frends are lnguss 2004 Mul-Modal User Ineracon II Zheng-Hua an

4 Large vocabulary speech recognon A Block Dagram Inpu Speech Fron-end Sgnal rocessng Feaure Vecors Lngusc Decodng and Search Algorhm upu Senence Speech Corpora Acousc Model ranng Acousc Models Lexcon Language Model Language Model Consrucon ex Corpora Example Inpu Senence hs s speech Acousc Models h-h-s-h-z-s-p-h-ch Lexcon h-h-s hs h-z s s-p-y-ch speech Language Model hs s speech Lexcal Knowledge-base Grammar hs s hs speech hs s w w - b-gram language model w w - w -2 r-gram Mul-Modal language User modelec L.S. Lee 2007 Ineracon II Zheng-Hua an Key componens of LVCSR sysem Acousc models Language models Speech Feaure exracon Decoder search Words Applcaon Speech recognon nvolves: How o represen he sgnal How o model boh acousc and language consrans How o search for he opmal answer Mul-Modal User Ineracon II Zheng-Hua an

5 ar II: Speech sgnal represenaon Inroducon Speech sgnal represenaon emplae based approach DW Sascal model based approach HMM Varably Mul-Modal User Ineracon II Zheng-Hua an Shor-me processng soluon Assumng ha speech has non-me-varyng properes fxed excaon and vocal rac whn shor nervals rocessng shor segmens frames of he speech sgnal each me f x n m x m w n m Mul-Modal User Ineracon II Zheng-Hua an

6 Wndows Recangular wndow w[ n] 0 n Hammng wndow commonly used 2πn w[ n] cos 0 n Mul-Modal User Ineracon II Zheng-Hua an 2008 Choce of wndow Wndow ype Bandwdh of Hammng wndow s abou wce he bandwdh of Recangular Aenuaon of more han 40dB for Hammng as compared wh 4 db for Recangular ousde passband Wndow duraon Increase decrease wndow bandwdh should be larger han a pch perod bu smaller han a sound duraon Mul-Modal User Ineracon II Zheng-Hua an

7 Dmenson & speech represenaon he curse of dmenson he compuaonal cos ncreases exponenally wh he dmenson of he problem he frame-based analyss yelds a seuence as a new represenaon of he speech sgnal samples a 8000/sec vecors a 00/sec Mul-Modal User Ineracon II Zheng-Hua an Fron-end feaure exracon MFCC Mul-Modal User Ineracon II Zheng-Hua an

8 ar III: emplae based approach Inroducon Speech sgnal represenaon emplae based approach DW Sascal model based approach HMM Varably Mul-Modal User Ineracon II Zheng-Hua an emplae based ASR x unknown speech Feaure Exracon X feaure vecor seuence aern Machng Decson Makng W oupu word y ranng speech Feaure Exracon Y Reference aerns emplae machng mechansm Calculae he dsance beween wo paerns Dynamc me warpng DW Mul-Modal User Ineracon II Zheng-Hua an

9 Speakng rae and me-normalzaon Speakng rae varaon causes nonlnear flucuaon n a speech paern me axs me-normalzaon s needed. Mul-Modal User Ineracon II Zheng-Hua an D based me-normalzaon Dynamc programmng s a paern machng algorhm wh a nonlnear me-normalzaon effec. me dfferences bw wo speech paerns are elmnaed by warpng he me axs of one so ha he maxmum concdence s aaned wh he oher also called dynamc me warpng DW he me-normalzed dsance s calculaed as he mnmzed resdual dsance beween hem remanng sll afer elmnang he mng dfferences. Mul-Modal User Ineracon II Zheng-Hua an

10 Dynamc programmng Consder wo speech paerns expressed as a seuence of feaure vecors : ck A a a2... a... ai B b b... b... b 2 Consder an - plane hen me dfferences can be depced by a seuence of pons c: where F c c2... c k... c K c k k k J Mul-Modal User Ineracon II Zheng-Hua an Dynamc programmng con d he seuence c s called a warpng funcon. A dsance bw wo feaure vecors s d c d a b he weghed summaon of dsances on warpng funcon F becomes E F K d c k. w k he me-normalzed dsance bw A and B s defned as he mnmum resdual dsance bw hem K d c k. w k k D A B mn K w k k k Mul-Modal User Ineracon II Zheng-Hua an

11 Resrcons on warpng funcon Warpng funcon F or pons ck as a model of me-axs flucuaon n speech has resrcons: Monoonc condons : k k and k 2 Connuy condons : k k and k 3 Boundary condons : and K I K J. 4 Adusmen wndow condon k k r 5 Slope consran condon : k k A graden should be neher oo seep nor oo genle. Mul-Modal User Ineracon II Zheng-Hua an he smples D of symmerc form Sep : Inalsaon: g 2d Sep 2: Ieraon D euaon: g + d g mn g + 2d g + d Adusmen wndow: r + r Sep 3: ermnaon: me-normalsed dsance D A B g I J where I + J Mul-Modal User Ineracon II Zheng-Hua an

12 From emplae o sascal mehod he emplae mehod wh D algnmen s a smplfed non-paramerc mehod whch s hard o characerse he varaon among uerances Hdden Markov model HMM s a powerful sascal mehod of characersng he observed daa samples of a dscree-me seres he underlyng assumpon of he HMM s he speech sgnal can be well characersed as a paramerc random process he parameers of he sochasc process can be esmaed n a precse well-defned manner Mul-Modal User Ineracon II Zheng-Hua an ar IV: Hdden Markov model Inroducon Speech sgnal represenaon emplae based approach DW Sascal model based approach HMM Varably Mul-Modal User Ineracon II Zheng-Hua an

13 Hdden Markov model Mul-Modal User Ineracon II Zheng-Hua an he Urn-and-Ball model doubly sochasc sysems Mul-Modal User Ineracon II Zheng-Hua an

14 Elemens of a dscree HMM : he number of saes saes s {s s 2...s } sae a me s M: he number of observaon symbols observaon symbols v {v v 2...v M } observaon a me o v A {a }: sae ranson probably dsrbuon a + s s B {b k}: observaon probably dsrbuon n sae b k v k s k M π π } { : nal sae dsrbuon For convenence we use he noaon: λ A B π Mul-Modal User Ineracon II Zheng-Hua an hree basc HMM problems Scorng: Gven an observaon seuence {o o 2...o } and a model λ {A Bπ} how o compue λ he probably of he observaon seuence? he Forward-Backward Algorhm Machng: Gven an observaon seuence {o o 2...o } and he model λhow o choose a sae seuence { 2... } whch s opmum n some sense? he Verb Algorhm ranng: How o adus he model parameers λ {ABπ} o maxmze λ? he Baum-Welch Re-esmaon rocedures Mul-Modal User Ineracon II Zheng-Hua an

15 roblem : Scorng Gven {o o 2...o } and λ {A Bπ} how o compue λ he probably of he observaon seuence? probably evaluaon Consder all possble sae seuences of lengh : λ λ λ Calculaon reured 2 all 2... π b a 2 For compuaons! b 2... a 2 b Mul-Modal User Ineracon II Zheng-Hua an he forward algorhm Consder he forward varable α defned as α oo2... o λ.e. he probably of he paral observaon seuence unl me and sae a me gven he model λ We can solve for α nducvely as follows:. Inalsaon α π b o 2. Inducon 3. ermnaon α α a b o + + λ α Calculaon 2.. For nsead of 0 72 Mul-Modal User Ineracon II Zheng-Hua an

16 Illusraon of forward algorhm Rabner 989 Mul-Modal User Ineracon II Zheng-Hua an he backward algorhm Smlarly consder he backward varable defned as β o o... o.e. he probably of he paral observaon seuence from me + o he end gven sae a me and model λ We can solve for β nducvely as follows:. Inalsaon β 2. Inducon λ β ab o 3. ermnaon λ π b o β Agan calculaon β + β 2... Mul-Modal User Ineracon II Zheng-Hua an

17 7 Mul-Modal User Ineracon II Zheng-Hua an roblem 2: Machng Gven {o o 2...o }how o choose a sae seuence { 2... } whch s opmum n some sense? pmal sae seuence rells dagram for an Isolaed Word Recognon ask. Mul-Modal User Ineracon II Zheng-Hua an Fndng opmal sae seuence ne opmaly creron s o choose he saes ha are ndvdually mos lkely a each me Defne he probably of beng n sae a me gven he observaon seuence and he model λ he ndvdually mos lkely sae * a me s λ γ λ λ λ λ Snce β λ α We have β α β α γ ] arg max[ * γ

18 Fndng opmal sae seuence con d he ndvdual opmaly creron has he problem ha he opmum sae seuence may no obey sae ranson consrans he opmal sae seuence may no even be a vald seuence a 0 for some and Anoher opmaly creron s s o fnd he sngle bes sae seuence pah.e. o maxmze λ he Verb algorhm a mehod based on dynamc programmng Mul-Modal User Ineracon II Zheng-Hua an he Verb algorhm o fnd he bes pah { 2... } for gven {o o 2...o } we defne he bes score hghes probably along a sngle pah a me δ max 2... oo2... o 2... λ whch accouns for he frs observaons and ends n sae. hen δ [maxδ a ]. b o + + Mul-Modal User Ineracon II Zheng-Hua an

19 9 Mul-Modal User Ineracon II Zheng-Hua an he Verb algorhm con d. Inalsaon 2. Recurson 3. ermnaon 4. ah sae seuence backrackng 0 o b ψ π δ a o b a 2 ] arg max[ 2 ] max[ δ ψ δ δ ] arg max[ ] max[ * * δ δ 2... * * + + ψ Mul-Modal User Ineracon II Zheng-Hua an he Verb algorhm con d Joseph cone

20 he power of recursve euaon Compung facorals n! Mehod. smply caculae n!for each n Mehod 2. use n! n n! f F n n!hen F n nf n for n Recursve Euaon Mul-Modal User Ineracon II Zheng-Hua an roblem 3: ranng How o une he model parameers λ {ABπ} o maxmze λ? - a learnng problem o effcen algorhm for global opmsaon Effecve erave algorhm for local opmsaon: he Baum-Welch re-esmaon Baum-Welch forward-backward algorhm Baum 972 s a specal case of EM expecaon-maxmzaon algorhm compues probables usng curren model λ; refnes λ o λ such ha λ s locally maxmsed uses α and β from forward-backward algorhm Mul-Modal User Ineracon II Zheng-Hua an

21 2 Mul-Modal User Ineracon II Zheng-Hua an Baum-Welch re-esmaon Defne he probably of beng n sae a me and sae a me + gven λ and.e. ξ b a b a b a β α β α λ β α λ λ λ ξ o o o Mul-Modal User Ineracon II Zheng-Hua an Baum-Welch Re-esmaon con d Recall ha s defned as he probably of beng n sae a me gven he enre observaon seuence and he model so Sum and over we have + ξ λ λ γ γ γ ξ n o sae ransons from sae expeced number of vsed. s ha sae mes he expeced number of n ransons from sae expeced number of ξ γ γ

22 22 Mul-Modal User Ineracon II Zheng-Hua an Baum-Welch re-esmaon formulas oherwse 0 mes n sae expeced number of and observng symbol mes n sae expeced number of ransons from sae expeced number of o sae ransons from sae expeced number of a me mes n sae expeced freuency number of.. k k k v o s k v o v o v o v k b a k δ λ δ λ γ γ γ ξ γ π Mul-Modal User Ineracon II Zheng-Hua an ar VI: Varably Inroducon Speech sgnal represenaon emplae based approach DW Sascal model based approach HMM Varably

23 Varably n he speech sgnal Mos noceable facors ha deermne accuracy are varaons n conex n speaker and n envronmen. Speech recognser can be very accurae for a parcular speaker n a parcular language and speakng syle n a parcular envronmen and lmed o a parcular ask. Bu remans a research challenge o buld a recognser ha can undersand anyone s speech n any language on any opc n any free-flowng syle and n any speakng envronmen Accuracy and robusness are he ulmae measures for he success of ASR Mul-Modal User Ineracon II Zheng-Hua an Varably Conex varably I s easy o recognse speech. I s easy o wreck a nce beach. Syle varably Isolaed connuous sponaneous Speaker varably human vocal rac Speaker-dependen vs. speaker-ndependen Speaker-adapaon Envronmenal varably Mulsyle ranng ransmsson channel varably Error concealmen Mul-Modal User Ineracon II Zheng-Hua an

24 roblems wh nose Hgh-level performance n conrolled envronmens Degradaon n nosy suaons 00% o 30% accuracy n a car wh 90km/h 99% o 50% n a cafeera Key ssue: msmach n ranng and operang envronmens Mul-Modal User Ineracon II Zheng-Hua an ose robusness Speech enhancemen x unknown speech + nose2 SR2 y ranng speech + nose SR Feaure Exracon Feaure Exracon X ose ressance Y Language models Decoder Acousc models Mul-condon ranng Mul-Modal User Ineracon II Zheng-Hua an W Model compensaon 24

25 ar VI: Summary Inroducon Speech sgnal represenaon emplae based approach DW Sascal model based approach HMM Varably Mul-Modal User Ineracon II Zheng-Hua an