What We re Talking About Today. Lecture 8. Part I. Review. LVCSR Training

Transcription

1 Wht We re Tlking out Tody Lecture 8 LVCSR Trining nd Decoding Michel Picheny, Bhuvn Rmhdrn, Stnley F. Chen Lrge-voculry continuous speech recognition (LVCSR). coustic model trining. How to estimte prmeters, e.g., for GMM s. How to uild phonetic decision trees. Decoding. How to select est word sequence... Given udio smple. IBM T.J. Wtson Reserch Center Yorktown Heights, New York, US {picheny,huvn,stnchen}@us.im.com 12 Novemer / 120 Prt I LVCSR Trining Review x Oservtions; sequence of 0d feture vectors. ω word sequence. Fundmentl eqution of SR. ω = rg mx ω P(ω x) = rg mx ω P(ω)P ω (x) P ω (x) coustic model. For word sequence ω, how likely re fetures x? P(ω) lnguge model. How likely is word sequence ω? / 120 / 120

2 Review: coustic Modeling For word sequence ω, construct ssocited HMM. g 1 /0.5 g 2 /0.5 g /0.5 g /0.5 g 5 /0.5 g 6 /0.5 g 1 /0.5 g 2 /0.5 g /0.5 g /0.5 g 5 /0.5 g 6 /0.5 Ech x cn e output y mny pths through HMM. Compute P ω (x) y summing over pth likelihoods. P ω (x) = P ω (x, ) pths Compute pth likelihood y... Multiplying rc nd GMM output pros long pth. coustic Likelihoods: Smll Voculry P ω (x) = P ω (x, ) pths = pths = T p t P( x t t ) t=1 T p t p t,j pths t=1 comp j dim d N (x t,d ; µ t,j,d, σ 2 t,j,d) p trnsition proility for rc. p,j mixture weight, jth component of GMM on rc. µ,j,d men, dth dim, jth component, GMM on rc. σ 2,j,d vrince, dth dim, jth component, GMM on rc. 5 / / 120 coustic Likelihoods: Lrge Voculry P ω (x) = P ω (x, ) pths = pths = T p t P( x t t ) t=1 T p t p t,j pths t=1 comp j dim d N (x t,d ; µ t,j,d, σ 2 t,j,d) So, Wht s Different for Lrge Voculry? The HMM. p trnsition proility for rc. p,j mixture weight, jth component of GMM on rc. µ,j,d men, dth dim, jth component, GMM on rc. σ 2,j,d vrince, dth dim, jth component, GMM on rc. 7 / / 120

3 Where re We? 1 coustic Modeling for LVCSR Review: Building HMM s, Smll Voculry Trining. Enumerte possile word sequences given trnscript. Replce ech word with its HMM; collect FB counts. 2 The Locl Mxim Prolem HMM eight HMMtwo Recipes for LVCSR Trining Discussion Decoding. Enumerte possile word sequences. Replce ech word with its HMM; run Viteri. HMMone HMM two HMM three 9 / / 120 Exmple: Word Models (Trining HMM) One HMM per word (two sttes per phone, sy). Ech HMM hs own GMM s (one per stte). e.g., reference trnscript: EIGHT TWO. EY TD T UW Wht s the Prolem With Word Models? Wht if wnt to e le to decode... Word not in trining set, e.g., REDONKULOUS? Lots of dt for some words. lmost no dt for others. Not sclle to lrge voculry. HMM eight HMMtwo g eight,1 g eight,2 g eight, g eight, g two,1 g two,2 g two, g two, g eight,1 g eight,2 g eight, g eight, g two,1 g two,2 g two, g two, 11 / / 120

4 Phonetic Modeling One HMM per phoneme. Ech HMM hs own GMM s. Need pronuncition or seform for ech word. TWO T UW TEN T EY N Conctente phoneme HMM s to form HMM for word. i.e., shre GMM s for phone cross ll words... Contining tht phone. Wht if word not in trining? No prolemo. Wht if phoneme not in trining? Unlikely. Phonetic Modeling TWO T UW EIGHT EY TD HMM EY HMM TD HMM T HMM UW g EY,1 gey,2 g TD,1 g TD,2 g T,1 g T,2 g UW,1 g UW,2 g EY,1 g EY,2 g TD,1 g TD,2 g T,1 g T,2 g UW,1 g UW,2 1 / / 120 Wht s the Difference? Pop Quiz g eight,1 g eight,2 g eight, g eight, g two,1 g two,2 g two, g two, g eight,1 g eight,2 g eight, g eight, g two,1 g two,2 g two, g two, g EY,1 gey,2 g TD,1 g TD,2 g T,1 g T,2 g UW,1 g UW,2 g EY,1 g EY,2 g TD,1 g TD,2 g T,1 g T,2 g UW,1 g UW,2 HMM topology typiclly doesn t chnge. HMM prmeteriztion chnges. Scenrio: 1000 word voculry; 50 phonemes. vg. word length = 5 phones; two sttes per phoneme. Word modeling: one HMM per word. How mny GMM s per word on verge? How mny GMM s in whole system? Phonetic modeling: one HMM per phoneme. How mny GMM s per phoneme? How mny GMM s in whole system? 15 / / 120

5 Context-Independent Phonetic Modeling Sme phoneme HMM independent of phonetic context. Wht s the prolem? Is L in S L IH nd IH L Z the sme? llophonic vrition; corticultion. Symptom: too few GMM s underfitting. Context-Dependent Phonetic Modeling Seprte HMM for ech context of ech phoneme? e.g., triphone model context is ± 1 phone. Seprte HMM for L-S+IH, L-IH+Z,... Wht s the prolem? Solution: cluster triphones. e.g., L-S+IH, L-S+, L-S+E, L-S+EH,... Seprte HMM for ech cluster. Most populr method: decision trees. 17 / / 120 Exmple: Tree for Phoneme T pos -1 S TS Z Y pos +1 HMM T,1 XR ER R N pos +1 Y X XR B HMM T,2 BD CH D... UW... N Y pos +1 HMM T, IH IX IY N Y HMM T, HMM T,5 N How Mny Trees? Which phoneme position ffects pronuncition of... Beginning of current phoneme the most? Wht out end of current phoneme? Seprte decision tree for ech phoneme HMM stte! If 50 phones, 2 sttes/phone, how mny trees totl? For ech tree, one GMM per lef. HMM topology fixed. Choose GMM to use t ech position... By finding lef in corresponding tree. 19 / / 120

6 Exmple: Tree for Phoneme T, Stte 2 Context-Dependent Phonetic Modeling gt.2,1 Y pos -1 S TS Z gt.2,2 Y N pos +1 XR ER R N pos +1 X XR B BD CH D... UW... N Strt with phoneme sequence. HMM EY HMM TD HMM T HMM UW Sustitute in HMM topology for ech phoneme. g EY.1 gey.2 g TD.1 g TD.2 g T.1 g T.2 g UW.1 g UW.2 g EY.1 g EY.2 g TD.1 g TD.2 g T.1 g T.2 g UW.1 g UW.2 Select GMM for ech stte using ssocited tree. g EY.1,? gey.2,? g TD.1,? g TD.2,? g T.1,? g T.2, g UW.1,? g UW.2,? gt.2, Y gt.2, Y pos +1 IH IX IY N gt.2,5 g EY.1,? g EY.2,? g TD.1,? g TD.2,? g T.1,? g T.2, g UW.1,? g UW.2,? 21 / / 120 Pop Quiz Scenrio: 1000 word voculry; 50 phonemes. vg. word length = 5 phones; two sttes per phoneme. Ech decision tree contins 100 leves on verge. Word modeling: one HMM per word. How mny GMM s per word on verge? 10. How mny GMM s in whole system? 10,000. Phonetic modeling, CI: one HMM per phoneme. How mny GMM s per phoneme? 2. How mny GMM s in whole system? 100. Phonetic modeling, CD: mny HMM s per phoneme. How mny GMM s per phoneme? How mny GMM s in whole system? Size Mtters Typicl model sizes: GMM s/ type HMM stte GMM s Gussins word per word k CI phone per phone k k CD phone per phone k 10k 10k 00k 0d feture vectors 80 prmeters/gussin. Big models cn hve tens of millions of prmeters. 2 / / 120

7 Recp Word modeling doesn t scle. Don t shre dt etween words. Some words hve lots of dt; other very little. Cn t model corticultion cross words. Phonetic modeling scles. Shre dt etween words; prmeter tying. Every phoneme hs lots of dt... But some lots more thn others. Context-dependent phonetic modeling. Models corticultion, including cross-word. More dt more leves more prmeters. Cn spred dt evenly cross GMM s. Discussion CD phonetic modeling with decision trees. Stte of the rt since erly 1990 s. No serious chllenger on horizon? triphone model ±1 phones of context. quinphone model ±2 phones of context. Longer context mkes decoding much hrder! Bsic issue: prmeter tying. Ech stte for ech phoneme hs own decision tree. Ech lef in ech decision tree hs own GMM. Shre lef GMM cross ll words contining lef. Wht re other possile schemes? 25 / / 120 Where re We? 1 coustic Modeling for LVCSR 2 The Locl Mxim Prolem Recipes for LVCSR Trining Trining Prmeter Estimtion Likelihood of trining dt is function of prmeter vlues. Trnsition proilities. GMM s: mixture weights; mens nd vrinces. Find prmeter vlues to mximize likelihood. Tool: Forwrd-Bckwrd lgorithm. Given initil vlues, itertively djust prmeters... To improve likelihood. i.e., find closest locl mximum to strt. Discussion 27 / / 120

8 Smll Voculry Trining L 2 Phse 1: Flt strt. Initilize ll Gussin mens to 0, vrinces to 1. Phse 2: Run Forwrd-Bckwrd lgorithm to convergence. Phse : Profit! Lrge Voculry Trining Wht s chnged? L 2: <2500 prmeters. Lrge voculry: up to 10M+ prmeters. Relisticlly, cn t do simple hill-climing serch... On 10M+ prmeters nd find good locl mximum. It s mircle it works with 2500 prmeters. 29 / / 120 Hill Climing nd Locl Mxim FB finds nerest mximum to initil prmeters. With d strting point, finl model will e grge. How to find good strting point? Where Do Locl Mxim Come From? ML estimtion for non-hidden models is esy. e.g., non-hidden HMM s; Gussins; multinomils. Count nd normlize; no serch necessry. Prolem must e hidden vriles! likelihood likelihood prmeter vlues prmeter vlues 1 / / 120

9 Wht re The Hidden Vriles? Hidden Vriles nd Locl Mxim P ω (x) = pths T p t p t,j t=1 comp j dim d N (x t,d ; µ t,j,d, σ 2 t,j,d) ssume ech GMM hs single component not hidden. Let s ssign vlues to every hidden vrile... In whole trining set. i.e., which GMM genertes ech frme. Look for sums or mx s. Pth through HMM which GMM/stte t ech frme. Which component in ech GMM t ech frme. g EY.1 gey.2 g TD.1 g TD.2 g T.1 g T.2 g UW.1 g UW.2 g EY.1 g EY.2 g TD.1 g TD.2 g T.1 g T.2 g UW.1 g UW.2 frme GMM EY.1 EY.1 EY.2 EY.2 EY.2 TD.1... Cll hidden ssignment over whole corpus n lignment. / 120 / 120 lignments nd Prmeter Initiliztion Fixing lignment mking corpus non-hidden. Esy to do ML estimtion of prmeters. Like Viteri-style trining in L 2. i.e., cn use lignment to initilize prmeters. Dt used to trin given GMM comes from... ll frmes ligned to tht GMM. If seed prmeters using d lignment... Wrong dt used to trin GMM s. Prmeters ner d mximum? If seed prmeters using good lignment... Right dt used to trin GMM s. Prmeters ner good mximum? Exmple: Good nd Bd lignments g EY.1 gey.2 g TD.1 g TD.2 g T.1 g T.2 g UW.1 g UW.2 g EY.1 g EY.2 g TD.1 g TD.2 g T.1 g T.2 g UW.1 g UW.2 Good lignment mtches truth. GMM models wht it s supposed to e modeling. e.g., GMM ssocited with first stte of TD-EY+T... ligns to initil frmes of TD in this context. frme truth EY.1 EY.1 EY.2 EY.2 EY.2 TD.1... hyp EY.1 EY.1 EY.2 EY.2 EY.2 TD.1... Bd lignment doesn t mtch truth. frme truth EY.1 EY.1 EY.2 EY.2 EY.2 TD.1... hyp EY.1 EY.2 EY.2 TD.1 TD.1 TD / / 120

10 Prmeter Initiliztion Key to finding good strting point for FB: Need good lignment to seed prmeters! Point: if hve existing good model... Use model to compute (Viteri) lignment. Use lignment to ootstrp nother model. Repet to uild more nd more complex models! Where to get first good model? Where does FB with flt strt ctully work!? Build lots of incrementlly more complex models... Or go stright from initil model to finl model? The Bsic Pln Step 1: Build CI model with 1 Gussin/GMM. Know flt strt + FB works! Step 2: Build CI model with 2 Gussins/GMM. Seed using lignment from lst system; run FB Step k: Build CD model with 128 Gussins/GMM. Seed using lignment from lst system; run FB. 7 / / 120 Wys to Seed Next Model From Lst One Vi lignment. Do Viteri-style trining for next model... Using Viteri lignment computed using lst model. frme GMM EY.1 EY.1 EY.2 EY.2 EY.2 TD.1... Vi prmeters. Seed prmeters of next model so... Viteri lignment is sme (or close) s for lst model. e.g., GMM splitting (clone ech Gussin, pertur). e.g., CI CD GMM s (clone ech CI GMM). Recp For models with millions of prmeters... Flt strt nd FB just doesn t cut it. Locl mxim due to hidden vriles. i.e., spce of possile lignments. If hve good lignment... Cn initilize prmeters so ner good mximum. Key ide: use simple models to ootstrp... Incrementlly more complex models. More gory detils to follow. 9 / / 120

11 Where re We? 1 coustic Modeling for LVCSR 2 The Locl Mxim Prolem Overview of Trining Process Strt: CI, GMM s contin single component. End: CD, GMM s contin 128 components, sy. How to get here from there? More thn one wy. Let s go through one recipe, strt to finish. Recipes for LVCSR Trining Discussion 1 / / 120 Step 0: Prerequisites Dt. Utternces with trnscripts. Pronuncition/seform dictionry. Questions to sk in phonetic decision tree. Decisions. For ech phoneme, HMM topology/size. Numer of components in GMM s. Period. The Pronuncition Dictionry Need pronuncition of every word in trining dt. Without pronuncition, cn t uild HMM for word. Words my hve multiple pronuncitions. THE(01) DH H THE(02) DH IY Where to get seforms for new words? sk linguist? (We fired them.) Where else? / 120 / 120

12 Step 1: CI, 1 component/gmm Flt strt. Trnsition proilities, mixture weights uniform. Gussin mens 0, vrinces 1. Run FB to convergence (L 2). Before: lignments re grge. fter: lignments re resonle (ut flwed). Step 2: CI, 2 components/gmm Split Gussins 2 components/gmm. Run unch of itertions of FB. Split Gussins components/gmm. Run unch of itertions of FB. Split Gussins 8 components/gmm. Run unch of itertions of FB. Split Gussins 16 components/gmm. Run unch of itertions of FB. 5 / / 120 Exmple: Gussin Splitting Trin single Gussin vi Forwrd-Bckwrd. Exmple: Gussin Splitting Split ech Gussin in two (±0.2 σ) 7 / / 120

13 Exmple: Gussin Splitting Run FB for few itertions. Exmple: Gussin Splitting Split ech Gussin in two (±0.2 σ) 9 / / 120 Exmple: Gussin Splitting Run FB for few itertions. There is lso k-mens Use centers s mens of Gussins; trin / / 120

14 The Finl Mixtures, Splitting vs. k-mens Step : Select Pronuncition Vrints Reference trnscript doesn t tell you everything. Missing silence, filled puses (e.g., UH). Doesn t tell you which pronuncition... For words with multiple pronuncitions. e.g., whether THE pronounced DH H or DH IY. THE(01) DH H THE(02) DH IY / / 120 Hndling ll Possile lterntives In theory, optionl silence, multiple pronuncitions... No prolem! Just uild pproprite HMM. Consider ll possile pths over whole trining process. ~SIL(01) THE(01) THE(02) ~SIL(01) DOG(01) DOG(02) DOG(0) ~SIL(01) In prctice, pinful. Expensive computtionlly. Building trining HMM with CD models tricky. Wht To Do? Solution: nil down exct trnscript. ~SIL(01) THE(01) DOG(02) ~SIL(01) Once model sufficiently good, compute Viteri pth. Identify pronuncitions (nd silences) long pth. Fix exct trnscript for reminder of trining. Or recompute periodiclly. 55 / / 120

15 Step : Select Pronuncition Vrints Run Viteri lgorithm on trining set. Compute exct trnscript for ech utternce. Run unch of itertions of FB. Step : Building Phonetic Decision Trees Gol: uild phonetic decision tree... For ech stte in ech phone HMM ( 150 totl). e.g.,.1,.2,., E.1,... Wht do we need? Dt ligned to ech phone HMM stte. List of cndidte questions. 57 / / 120 Trining Dt for Decision Trees Run Viteri lgorithm. For ech frme, identify which feture vector,... Which GMM/HMM stte, nd phonetic context. g EY.1 gey.2 g TD.1 g TD.2 g T.1 g T.2 g UW.1 g UW.2 g EY.1 g EY.2 g TD.1 g TD.2 g T.1 g T.2 g UW.1 g UW.2 frme GMM EY.1 EY.1 EY.2 EY.2 EY.2 TD.1... Building (Triphone) Tree Input: list of triples ( x i, p L,i, p R,i ). t ech node on frontier of tree: Choose question of form... Does phone in position j elong to set q?... Optimizing i P( x i lef(p L,i, p R,i ))... Where ech lef distriution is single Gussin. Cn efficiently uild whole level of tree in single pss. See Lecture 6 slides nd redings for gory detils. e.g., feture vector x 5 used to trin tree for TD.1. (Triphone) context is -EY+T. Dt for tree is list of triples ( x, p L, p R ); e.g., (x 5, EY, T). 59 / / 120

16 The List of Cndidte Questions Exmple Questions Creted y linguist mny decdes go. Pssed down from mother to dughter, fther to son. Corresponds to phonetic concepts. e.g., vowel? dipthong? frictive? nsl? etc. Ech question represented s set of phones. Does phoneme elong to set of not? E... ZH O OY X IH OW UW SH ZH S Z E EH EY B D G F HH TH CH JH K P T DH V M N NG ER R S TS Z F TH H O X EY IH IY CH JH SH ZH IY Y DH F TH V 61 / 120 L W / 120 Exmple Output Step : Building Phonetic Decision Trees pos +1 XR ER R N Build phonetic decision tree for ech phone stte. Before: one (CI) GMM per phone stte. fter: one (CD) GMM per lef for ech phone stte. Seed CD GMM s y cloning originl CI GMM. pos -1 B D G K P T Y pos -1 B BD CH D DD DH... Initilly, sme Viteri lignment s CI model. In computing likelihood, replce CI with CD GMM... But these re identicl. Run unch of itertions of FB. Y N Y N go.2,1 go.2,2 go.2, go.2, 6 / / 120

17 Step 5: CD, 128 components/gmm Split Gussins 2 components/gmm. Run unch of itertions of FB. Split Gussins 6 components/gmm. Run unch of itertions of FB. Split Gussins 128 components/gmm. Run unch of itertions of FB. Recp Step 0: Collect dt. Mke seforms for ll words in reference trnscripts. Step 1: Build CI, 1 component/gmm model from flt strt. Step 2: Build CI, mny component GMM model. Repeted Gussin splitting. Step : Find exct trnscripts, pronuncition vrints. Viteri lgorithm. Step : Build phonetic decision tree. From lignment creted y CI model. Step 5: Build CD, mny component GMM model. Repeted Gussin splitting. 65 / / 120 Discussion One of mny possile recipes. Trining is complicted, multi-step process. Motifs. Seed complex model using simpler model. Run lots of Forwrd-Bckwrd. Where re We? 1 coustic Modeling for LVCSR 2 The Locl Mxim Prolem Recipes for LVCSR Trining Discussion 67 / / 120

18 LVCSR Trining Doesn t Require Much Dt. Utternces with trnscripts. Pronuncition/seform dictionry. Questions to sk in phonetic decision tree. lgorithms. Viteri; Forwrd-Bckwrd. Decision-tree uilding. lmost sme s in smll voculry. Trining Is n rt Hidden model trining frught with locl mxim. Seed more complex models with simpler models. Incrementlly improve lignments; void d mxim. Recipes developed over decdes. Discovered vi swet nd ters. No one elieves these find glol mxim. How well recipe works depends on dt? 69 / / 120 Speeding Up Trining Requires mny, mny itertions of Forwrd-Bckwrd. Full Forwrd-Bckwrd trining. Compute posterior of ech lignment. Collect counts over ll possile lignments. Viteri-style trining. Pick single lignment, e.g., using Viteri. Collect counts over single lignment. Both vlid gurnteed to increse (Viteri) likelihood. When To Use One or the Other? Use Viteri-style when cn cheper. Optimiztion: need not relign every itertion. Intuitively, full FB my find etter mxim... But if posteriors very shrp, do lmost sme thing. Rememer exmple posteriors in L 2? Rule of thum: When first trining new model, use full FB. Once locked in to locl mximum, Viteri is fine. 71 / / 120

19 Bootstrpping One Model From nother Bootstrp complex model from simpler model... Using lignment computed from simpler model. Point: models need not e of sme form! Cn use WSJ model to ootstrp Switchord model. Cn use triphone model to ootstrp quinphone model. Cn use GMM/HMM model to ootstrp DBN model. Requirement: sme phonemes, sttes per phoneme. Whew, Tht Ws Pretty Complicted! The tip of the iceerg. dpttion (VTLN, fmllr, mmllr). Discrimintive trining (LD, MMI, MPE, fmpe). Model comintion (cross dpttion, ROVER). 7 / / 120 Things Cn Get Pretty Hiry How Long Does Trining Tke? 100-est rescoring VTLN MFCC 8.5% 9.% 7.7% 8.7% MMI-ST ML-ST-L ML-ST MMI-ST ML-ST-L ML-ST 1.6% 2.1% 0.9% 1.9% MMI-D 2.6% 5.6% MFCC-SI 8.1% 8.7% 6.7% ML-D-L ML-D MMI-D 0.% 1.0% 29.8% 5.9% Evl 98 WER (SWB only) 8.% Evl 01 WER ML-D-L ML-D 7.1% 100-est 8.1% 100-est 5.9% 100-est 6.9% 0.1% rescoring 0.5% rescoring 29.5% rescoring 0.1% PLP VTLN 1.6%.% 7.9% 0.8% It s secret. Mesure in terms of rel-time fctor. How mny hours to process one hour of speech? If 1,000 hours of speech, 10x rel time... How mny dys to trin on one mchine? Prlleliztion is key. Dt prlleliztion: collect FB counts on 1 th corpus. k Sum FB counts efore prmeter reestimtion. -grm rescoring -grm rescoring -grm rescoring -grm rescoring -grm rescoring Consensus Consensus Consensus Consensus Consensus -grm rescoring -grm rescoring 5.7% 29.2% -grm rescoring -grm rescoring -grm rescoring Consensus Consensus Consensus Consensus Consensus 6.5% 8.1% 7.2% 5.5% 5.2% 7.7% 6.% 29.9% 1.1% 0.2% 28.8% 28.7% 1.% 29.2% ROVER.0% 27.8% 75 / / 120

20 Recp In theory, trining involves simple lgorithms. In prctice, trining is insnely complicted... For stte-of-the-rt systems. dministrivi Cler (6); mostly cler (). Pce: OK (5), slow (2). Muddiest: dcs trees nd Gussins (2); dcs trees nd HMM s (2); criterion for constructing dcs trees (1). Feedck (2+ votes): More info on reding project (2). fll12/e6870/redings/project_f12.html (sme pssword s redings). Don t need to worry out this yet. 77 / / 120 dministrivi L 2, L. Not grded yet; hnded ck next lecture. nswers: /user1/fculty/stnchen/e6870/l2_ns/. L. Postponed ecuse mteril not covered yet. Will nnounce when l posted + new due dte. Mke-up lecture. Wht dys cn you mke it (sme time)? Working on setups for non-reding projects. Prt II Segue: Intro to LVCSR Decoding 79 / / 120

21 eight two nine one three four ve six zero seven Decoding for LVCSR Now know how to uild models for LVCSR: n-grm LM s P(ω) vi counting nd smoothing. CD coustic models P ω (x) vi complex recipes. This prt: given test udio x, how to compute... Most likely word sequence ω. ω = rg mx ω P(ω x) = rg mx ω Initilly, let s ignore efficiency. How to do this conceptully? P(ω)P ω (x) Decoding: Smll Voculry Tke (H)MM representing llowle word sequences/lm. one two three Replce ech word with corresponding HMM. HMMone HMM two HMM three Run Viteri lgorithm! 81 / / 120 Cn We Do Sme Thing for LVCSR? 1 Cn we express LM s (H)MM? 2 How to expnd word HMM to full HMM? Grph not too ig? Not too slow to decode? Issue 1: Is n-grm Model n (H)MM? Yup; n-grm model is Mrkov model of order n 1. Exmple: trigrm model P(w i w i 2 w i 1 ). One stte for ech history w i 2 w i 1. rrive here iff lst two words re w i 2, w i 1. Ech stte w i 2 w i 1 hs outgoing rc for every w i... To stte w i 1 w i with proility P(w i w i 2 w i 1 ). For ech word sequence w 1,..., w L... Single pth through HMM with totl proility P(w 1,..., w L ) = i P(w i w i 2 w i 1 ) 8 / / 120

22 Trigrm LM, Morse Code, Bsic Structure Trigrm LM, Morse Code, With Proilities /P (j ) /P (j ) /P (j ) /P (j ) /P (j ) /P (j ) /P (j ) /P (j ) 85 / / 120 Pop Quiz How mny sttes in HMM representing trigrm model... With voculry size V? How mny rcs? Issue 2: Grph Expnsion Trining: only single word sequence, e.g., EIGHT TWO. EY TD T UW EY-j+TD TD-EY+T T-TD+UW UW-T+j g EY.1,9 gey.2,2 g TD.1,6 g TD.2,7 g T.1,15 g T.2, g UW.1, g UW.2,1 g EY.1,9 g EY.2,2 g TD.1,6 g TD.2,7 g T.1,15 g T.2, g UW.1, g UW.2,1 87 / / 120

23 Context-Dependent Grph Expnsion Decoding: mny possile word sequences. CD expnsion: hndling rnch points is tricky. Other issues: single-phoneme words; quinphone models. Issue: How Big The Grph? Trigrm model (e.g., voculry size V = 2) T TWO UW ONE W H W-N+H UW-T+T N-H+W N-H+T T-N+UW T-UW+UW H-W+N UW-T+W W-UW+H N V word rcs in FS representtion. Sy words re phones = 12 sttes on verge. If V = 50000, sttes in grph. PC s hve ytes of memory. 89 / / 120 Issue: How Slow Decoding? In ech frme, loop through every stte in grph. If 100 frmes/sec, sttes... How mny cells to compute per second? PC s cn do floting-point ops per second. Recp: Smll vs. Lrge Voculry Decoding In theory, cn use sme exct techniques. In prctice, three ig prolems: Context-dependent grph expnsion is complicted. Decoding grphs wy too ig. Decoding wy too slow. How cn we hndle this? Next week: Finite-stte mchines. How to mke decoding efficient. 91 / / 120

24 eight two nine one three four ve six zero seven Prt III Finite-Stte Mchines View of Grph Expnsion Step 1: Tke word grph s input. Convert into phone grph. Step 2: Tke phone grph s input. Convert into context-dependent phone grph. Step : Tke context-dependent phone grph. Convert into finl HMM. Gol: wnt frmework for... 1 Representing grphs. 2 Trnsforming grphs. 9 / / 120 View of Grph Expnsion T TWO UW ONE W H N UW-T+T N-H+W N-H+T T-N+UW T-UW+UW Frmework for Rewriting Grphs How to represent grphs? HMM s finite-stte cceptors (FS s)! How to represent grph trnsformtions? Finite-stte trnsducers (FST s)! Wht opertion pplies trnsformtions to grphs? Composition! W-N+H H-W+N UW-T+W W-UW+H 95 / / 120

25 Where re We? 1 The Bsics 2 Composition Wht is Finite-Stte cceptor? It s like n HMM, ut without proilities. It hs sttes. Exctly one initil stte; one or more finl sttes. It hs rcs. Ech rc hs lel, which my e empty (ɛ). c 97 / / 120 Wht Does n FS Men? The (possily infinite) list of strings it ccepts. i.e., strings tht lel pth from initil to finl stte. Mening:,, c. Pop Quiz re these equivlent? i.e., do they hve sme mening? c Mening:,,,,... Things tht don t ffect mening. How lels re distriuted long pth. Invlid pths. 99 / / 120

26 Wht is Finite-Stte Trnsducer? It s like finite-stte cceptor, except... Ech rc hs two lels insted of one. n input lel (possily empty). n output lel (possily empty). c:c Wht Does n FST Men? (possily infinite) list of pirs of strings... n input string nd n output string. Mening: (, ), (, B), (c, C). : :B c:c : : : Mening: (ɛ, ɛ), (, ), (, ), (, ),... : : 101 / / 120 Wht is Composition? pplying FST T to FS to crete new FS T. If α nd (α, β) T, then β T. hs mening:,, c. T hs mening: (, ), (, B), (c, C). : c :B Recp Finite-stte cceptor (FS): one lel on ech rc. Finite-stte trnsducer (FST): two lels on ech rc. Finite-stte mchine (FSM): FS or FST. lso, finite-stte utomton. FST s cn e used to trnsform FS s vi composition. The point: cn express ech stge in grph expnsion... s pplying FST vi composition. c:c T hs mening:, B, C. B C 10 / / 120

27 Where re We? 1 The Bsics 2 Composition The Composition Opertion simple nd efficient lgorithm for computing... Result of pplying trnsducer to cceptor. Wht cn composition do? 105 / / 120 Rewriting Single String Single Wy Rewriting Single String Single Wy 1 2 d 1 2 d : 1 2 T :B d:d T d:d c:c :B : T B D 1 2 T B D 107 / / 120

28 Trnsforming Single String Let s sy hve string, e.g., THE DOG Let s sy wnt to pply one-to-one trnsformtion. e.g., mp words to their (single) seforms. DH H D O G This is esy, e.g., use sed or perl or... The Mgic of FST s nd Composition Let s sy hve (possily infinite) list of strings... Expressed s n FS, s this is compct. How to trnsform ll strings in FS in one go? How to do one-to-mny or one-to-zero trnsformtions? Cn we express (possily infinite) list of output strings... s (compct) FS? Fst? 109 / / 120 Rewriting Mny Strings t Once Rewriting Single String Mny Wys 1 T d:d c:c :B : 1 c d d 1 2 T :B : : : 1 2 T 1 B B T 1 C D 2 5 D B / / 120

29 Rewriting Some Strings Zero Wys Computing Composition: The Bsic Ide 1 d 2 5 For every stte s, t T, crete stte (s, t) T... Corresponding to eing in sttes s nd t t sme time. Mke rcs in intuitive wy. 6 : T 1 5 T / / 120 Exmple Computing Composition: More Formlly 1 2 For now, pretend no ɛ-lels. For every stte s, t T, crete stte (s, t) T. : 1 2 T 1, 2, :B, Crete rc from (s 1, t 1 ) to (s 2, t 2 ) with lel o iff... There is rc from s 1 to s 2 in with lel i nd... There is rc from t 1 to t 2 in T with lel i : o. T 1,2 2,2 B,2 (s, t) is initil iff s nd t re initil; similrly for finl sttes. (Remove rcs nd sttes tht re unrechle.) Wht is time complexity? 1,1 2,1,1 Optimiztion: strt from initil stte, uild outwrd. 115 / / 120

30 nother Exmple Composition nd ɛ-trnsitions 1 : 2 Bsic ide: cn tke ɛ-trnsition in one FSM... Without moving in other FSM. Tricky to do exctly right. Do redings if you cre: (Pereir, Riley, 1997) T 1 :B 2 : <epsilon> 1 2, T B 1 <epsilon>:b 2 : B:B : eps 1, 2,, B B B T 1,1 2,2,1 1,2 2,1,2 T 1,2 eps 2,2,2 B B B 1,1 eps 2,1,1 117 / / 120 Recp FST s cn express wide rnge of string trnsformtions. Composition lets us efficiently... pply FST to ll strings in FS in one go! FSM Toolkits T&T FSM toolkit OpenFST; lots of others. Implements composition, lots of finite-stte opertions. syntx for specifying FS s nd FST s, e.g., 1 2 C 2 B C 1 2 B 119 / / 120