Doctoral Course in Speech Recognition

Size: px

Start display at page:

Download "Doctoral Course in Speech Recognition"

Evangeline Warner
6 years ago
Views:

1 Docoral Course in Speech Recogniion Friday March 30 Mas Blomberg March-June 2007 March 29-30, 2007 Speech recogniion course 2007 Mas Blomberg

2 General course info Home page hp:// se_pm.hml Exercises VQ, CART, HMM decoding, raining Reurn soluions by May 7 Term paper 4-6 pages, max 0 Send o reviewers ( 2 course paricipans by May 6 Reviewer reurn commens by May 25 Final paper o eacher and he reviewers by June Closing seminar Presenaion of own paper Acive discussions March 29-30, 2007 Speech recogniion course 2007 Mas Blomberg 2

3 Day # Course overview Probabiliy, Saisics and Informaion Theory (pp 73-3: 59 pages Paern Recogniion (pp 33-97: 65 pages Speech Signal Represenaions (pp pages Hidden Marov Models (pp : 37 pages Day #2 Hidden Marov Models (con. Acousic Modeling (pp : 6 pages Environmenal Robusness (pp : 68 pages Compuaional exercise Day #3 Language Modeling (pp : 46 pages Basic Search Algorihms (pp : 53 pages Large-Vocabulary Search Algorihms (pp : 4 pages Applicaions and User Inerfaces (pp : 38 pages Oher opics Day #4 Closing seminar Presenaions of erm papers March 29-30, 2007 Speech recogniion course 2007 Mas Blomberg 3

4 8.2.4 How o Esimae HMM Parameers - Baum-Welch Algorihm The mos difficul of he hree HMM problems Unsupervised learning. Incomplee daa. Sae sequence unnown. Use he EM algorihm Implemened by he ieraive Baum-Welch (Forward- Bacward algorihm March 29-30, 2007 Speech recogniion course 2007 Mas Blomberg 4

5 The EM Algorihm for HMM raining Problem approached Esimae model parameers ha maximize he probabiliy ha he model have generaed he daa. The maximum is achieved when he model disribuion is equal o ha of he he raining daa Esimae disribuions (ML of several classes (model saes when he raining daa (ime frames are no classified Is i possible o rain he classes anyway? (Yes - local maximum only Ieraive procedure, simplified descripion. Iniialise class disribuions 2. Using curren parameers, compue he class (sae probabiliies for each raining sample (ime frame 3. Every class (sae disribuion is re-compued as a probabiliy weighed conribuion of each sample (ime frame A moving arge; he new disribuion will affec he class probabiliies, which will, in urn, resul in new parameers, herefore: 4. Repea 2+3 unil convergence (Will converge Similar principle for observaion and ransiion probabiliies March 29-30, 2007 Speech recogniion course 2007 Mas Blomberg 5

6 Simplified illusraion of EM esimaion for HMM raining Say, hree pahs have been found. The probabiliies of he sae sequences for he iniial HMM are 0.3, 0.35 and s P = 0.3 s 3 s 2 P = 0.35 P = 0.22 s 2 3 T New E(s 2 = (0.3 X( X ( X( 3 / 0.70 No as simple as i may loo. Many pahs, parly shared March 29-30, 2007 Speech recogniion course 2007 Mas Blomberg 6

7 Towards Baum-Welch algorihm No feasible o compue over all individual possible sae sequences And no necessary The probabiliy ha he model has aen a cerain ransiion a a cerain ime is independen of he hisory and he fuure (Marov assumpion We only need o now heir summed effec o he probabiliy for every individual ransiion in he rellis diagram (ime-sae P(The model swiches beween saes i and a ime March 29-30, 2007 Speech recogniion course 2007 Mas Blomberg 7

8 The Baum-Welch algorihm P(The model swiched from sae i o a ime = γ (i, a b a c d d a b Transiion probabiliy: HMM a 23 a 23 8 = = 8 4 = = 2 γ (2,3 γ (2, Uerance T γ 4 (2,3 March 29-30, 2007 Speech recogniion course 2007 Mas Blomberg 8

9 The Baum-Welch Algorihm - 2 γ (i,: The probabiliy of he model having aen he ransiion from sae i o sae a ime and produced he observaions = The sum of he probabiliies for all pahs passing hrough (-,iand (, divided by he sum of he probabiliies for all pahs α ( i a b ( X β ( i = N = T α ( T March 29-30, 2007 Speech recogniion course 2007 Mas Blomberg 9

10 Forward and Bacward probabiliies N Forward probabiliy α (i α ( = α ( i ai b ( X i= The probabiliy of generaing a parial observaion X X ending a ime and sae i Bacward probabiliy The probabiliy of generaing a parial observaion X + X T saring from ime and sae i. N β ( i = aib ( X = β (i β+ ( = T,..., i N T ( i = / N + β Compuaion of α and β for a linear, lef-righ model March 29-30, 2007 Speech recogniion course 2007 Mas Blomberg α ( i, β ( i 0 T

11 The Baum-Welch Algorihm (con. Def γ (i,: The probabiliy of he model having aen he ransiion from sae i o sae a ime γ ( i, = P(The model has swiched from sae i o a ime P(The model generaes he observed sequence and swiches from sae i o = P(The model generaes he observed sequence α ( i a b ( X β ( i = N = T α ( a ime March 29-30, 2007 Speech recogniion course 2007 Mas Blomberg

12 The Baum-Welch Algorihm (con. New model esimaes: aˆ i bˆ ( T = = T N = = X = T γ ( i, (8.40 γ ( i, γ ( i, = o i (8.4 γ ( i, = i Quie inuiive equaions! The raio beween he expeced number of ransiions from sae i o and he expeced number of all ransiions from sae i The raio beween he expeced number of imes he observaion daa emied from sae is o and he expeced number of imes any observaion daa is emied from sae March 29-30, 2007 Speech recogniion course 2007 Mas Blomberg 2

13 8.3 Coninuous and Semi-Coninuous HMMs The observaion does no come from a finie se, bu from a coninuous space No quanizaion error March 29-30, 2007 Speech recogniion course 2007 Mas Blomberg 3

14 The Baum-Welch algorihm Coninuous Mixure Densiy HMMs P(The model swiched o sae a ime using mixure componen = ζ (, x x + x +2 Coninuous observaions HMM Uerance T General idea: One arrow for each mixure componen March 29-30, 2007 Speech recogniion course 2007 Mas Blomberg + P(symbol => N(μ,σ 4

15 March 29-30, 2007 Speech recogniion course 2007 Mas Blomberg Coninuous Mixure Densiy HMMs A weighed sum of mulivariae Gaussians M: number of mixure-componens c : he weigh for he h componen in sae Similar re-esimaion equaions as he discree case bu an exra dimension is he probabiliy of each mixure componen having produced he observaion = Σ = M x N c b,, ( ( μ x = = M c = = = T T, (, ( ˆ ζ ζ x μ = = = Σ T T x x ', ( ˆ ( ˆ (, ( ˆ ζ ζ μ μ = = = = T M T c, (, ( ˆ ζ ζ = = = N i T N i i i x b c a i ( ( ( (, ( α β α ζ Zea: The probabiliy ha here is a ransiion o sae and mixure comp, a ime given ha he model generaes he observed sequence

16 Discree HMM Observaions are discree symbols, e.g. VQ codeword indeces VQ space Codeword probabiliies HMM saes March 29-30, 2007 Speech recogniion course 2007 Mas Blomberg 6

17 Coninuous HMM Coninuous observaions The mixure componens and he mixure weighs are sae-dependen Gauss componens Mixure weighs HMM saes March 29-30, 2007 Speech recogniion course 2007 Mas Blomberg 7

18 Semiconinuous (Tied-Mixure HMM Coninuous observaions The mixure componen disribuions are sae-independen The mixure weighs are sae-dependen -dim. Gauss componens common o all saes Sae-dependen mixure weighs HMM saes March 29-30, 2007 Speech recogniion course 2007 Mas Blomberg 8

19 8.3.2 Semiconinuous HMMs (SCHMM (Tied-mixure HMM Bridging he gap beween discree and coninuous mixure densiy HMMs. As in discree HMM, a common codeboo for all saes Bu coninuous pdfs of Gaussians - no discree symbols Sae-dependen mixure weighs corresponds o discree oupu probabiliies b The observaion probabiliy is a sum of he individual probabiliies for all mixure componens. M b Reduced number of parameers (vs coninuous bu sill allowing deailed acousic modeling (vs discree Similar re-esimaion as for coninuous HMM, excep saeindependen mixures ( x = b ( N( x, μ, Σ = March 29-30, 2007 Speech recogniion course 2007 Mas Blomberg 9

20 8.4 Pracical Issues in Using HMMs Iniial Esimaes Model Topology Training Crieria Deleed Inerpolaion Parameer Smoohing Probabiliy Represenaions March 29-30, 2007 Speech recogniion course 2007 Mas Blomberg 20

21 8.4. Iniial Esimaes Imporan in order o reach a high local maximum Discree HMM Iniial zero probabiliy remains zero Uniform disribuion wors reasonably well Coninuous HMM mehods -means clusering Proceed from discree HMM o semi-coninuous o coninuous Sar raining single mixure models. Use previously segmened daa or fla sar (equal model parameers of all saes in he raining daa March 29-30, 2007 Speech recogniion course 2007 Mas Blomberg 2

22 8.4.2 Model Topology Lef-o-righ - he mos popular opology Number of saes per model Phone: Three o five saes Shor words: 2-3 saes per phoneme Long words: -2 saes per phoneme Null ransiions Change sae wihou using observaions (e.g. iniial and final saes March 29-30, 2007 Speech recogniion course 2007 Mas Blomberg 22

23 8.4.3 Training Crieria Maximum Lielihood Esimaion (MLE Sensiive o inaccurae Marov assumpions MCE and MMIE migh wor beer MAP suiable for adapaion and small raining daa MLE he mos used simpliciy superior performance MCE and MMIE for small and medium vocabularies Combinaions possible March 29-30, 2007 Speech recogniion course 2007 Mas Blomberg 23

24 March 29-30, 2007 Speech recogniion course 2007 Mas Blomberg Deleed Inerpolaion Combine well-rained general models wih less well-rained deailed models The combined probabiliy is a weighed average (inerpolaion of he probabiliies of he separae models speaer-dependen and -independen models, conex-free and conexdependen phone models, unigrams, bigrams and rigrams V-fold cross-validaion o esimae λ Train on (v- pars, esimae λ on he remainding par, circulae EM algorihm: new λ = λ P(model A / (P(inerpolaed model DI ( ( ( ( x x x B A DI P P P λ λ + = ( ( (, (, (,, (, ( K C w C w C w w C w w C w w w C w w w P λ λ λ + + = Example: Combine rigrams, bigrams and unigrams

25 Alg. 8.5 Deleed Inerpolaion Procedure Sep : Iniialize λ wih a guessed esimae Sep 2: Updae λ by he formula: ˆ λ = λp ( x M ( x M n A = = λpa ( x + ( λ PB M: number of raining daa divisions P A- and P B- are esimaed on he all raining daa excep par n he number of daa poins in par aligned o he model x : he -h daa poin in he -h se Sep 3: Repea sep 2 unil convergence March 29-30, 2007 Speech recogniion course 2007 Mas Blomberg 25

26 8.4.5 Parameer Smoohing Compensae for insufficien raining daa Increase he daa (There is no daa lie more daa Reduce he number of free parameers Deleed inerpolaion Parameer flooring o avoid small probabiliy values Tying parameers (SCHMM Covariance marix inerpolae via MAP Tie marices Use diagonal covariance marices March 29-30, 2007 Speech recogniion course 2007 Mas Blomberg 26

27 8.4.6 Probabiliy Represenaions The probabiliies become very small underflow problem Vierbi decoding (only muliplicaion: use logarihm Forward-bacward (muliplicaion and addiion: difficul Soluion Scaling o mae i S α ( i = Soluion 2 Logarihmic probabiliies Use loo-up able o speed up log(p +P 2 March 29-30, 2007 Speech recogniion course 2007 Mas Blomberg 27

28 8.5 HMM Limiaions Duraion modeling Firs Order Assumpion Condiional Independence Assumpion March 29-30, 2007 Speech recogniion course 2007 Mas Blomberg 28

29 8.5. Duraion Modeling HMM duraion disribuion: exponenial decrease The probabiliy of sae duraion is he probabiliy of aing he self-loop imes muliplied by he probabiliy of leaving he sae d i ( = aii ( aii Differen o real disribuion (House & Crysal, 986 Maximum Lielihood search can model non-exponenial disribuions by sequence of saes Sandard Vierbi canno bu can be modified o a he expense of large increase in compuaion March 29-30, 2007 Speech recogniion course 2007 Mas Blomberg 29

30 Modified Vierbi for sae duraion modelling Algorihm τ V ( = Max Max V τ ( i aid ( τ b i τ l= Convenional Vierbi algorihm i [ ( i a b ( X ] V ( = Max V i ( X τ + l Maximize over previous sae i and saring ime -τ for end ime Large increase in complexiy O(D 2, (D = Max duraion Can be reduced (Seward, 2003 Modes accuracy improvemen March 29-30, 2007 Speech recogniion course 2007 Mas Blomberg 30

31 Recursion paern for explici duraion Vierbi decoding V ( τ = Max Max V τ ( i aid ( τ b i τ l= ( X τ + l HMM Explici duraion min = 2, max = 5 March 29-30, 2007 Speech recogniion course 2007 Mas Blomberg Sandard T 3

32 8.5.2 Firs Order Assumpion In a firs order Marov chain, he ransiion from a sae depends only on he curren sae A second order Marov chain models he ransiion probabiliy beween wo saes as dependen on he curren and he previous sae Transiion probabiliy: a s = P( s s s 2s s 2 Has no offered sufficien accuracy improvemen o usify he increase in compuaional complexiy March 29-30, 2007 Speech recogniion course 2007 Mas Blomberg 32

33 8.5.3 Condiional Independence Assumpion I is assumed ha all observaion frames are dependen only on he sae ha generaed hem, no on any oher frames in he observaion sequence (neighboring or disan Difficul o handle non-saionary frames wih srong correlaion To include he dependence on he previous observaion frame: T or P( X S, Φ = P( X X, s =, Φ T P( X S, Φ = P( X R( X, s =, Φ March 29-30, 2007 Speech recogniion course 2007 Mas Blomberg 33

.. fredag väll If he firs [e:] has cerain characerisics (due o accen, age, gender, ec.

34 Disan repeiions of idenical phonemes in an uerance are acousically similar a de är väl fredag idag... så de blir väl ehh... fredag väll If he firs [e:] has cerain characerisics (due o accen, age, gender, ec. hen i is liely ha he second one has i as well. Dependence! Problem in speaer-independen ASR March 29-30, 2007 Speech recogniion course 2007 Mas Blomberg

35 9.6 Adapive Techniques Minimizing Mismaches There is always mismach beween raining and recogniion condiions Adapaion Minimize he mismach dynamically wih lile calibraion daa Supervised nowledge of he correc ideniy Unsupervised he recogniion resul is assumed o be correc March 29-30, 2007 Speech recogniion course 2007 Mas Blomberg 35

36 9.6. Maximum a Poseriori (MAP A new model is esimaed using he raining daa inerpolaed wih old informaion abou he model T τ iμnw + ζ ( i, x i = ˆ μi = T τ + ζ ( i, τ i is a balancing facor beween he prior mean and he ML esimae. Can be a consan for all Gaussian componens Similar for he covariance esimaion Limiaions The prior model needs o be accurae Needs observaions for all models i = March 29-30, 2007 Speech recogniion course 2007 Mas Blomberg 36

37 9.6.2 Maximum Lielihood Linear Regression (MLLR Linear regression funcions ransform mean and covariance for maximizing he lielihood of he adapaion daa μ = A μ + b i c i A c is a regression marix, b c is an addiive vecor for regression class c A and b can be esimaed in a similar way as when raining he coninuous observaion parameers Ieraion for opimizaion Models no in he adapaion daa are updaed If lile raining daa, use few regression classes Can adap boh means and variances Does no adap ransiion probabiliies c March 29-30, 2007 Speech recogniion course 2007 Mas Blomberg 37

38 MLLR adapaion illusraion The ransform for a class is opimized o maximize he lielihood of he adaped models o generae he adapaion daa Original models Adaped models X M M M M M M M M M X X X Adapaion daa Adaped regression class Transform M M MM M M M M M Non-adaped regression class March 29-30, 2007 Speech recogniion course 2007 Mas Blomberg 38

39 Speaer-Adapive Training (SAT Problem Large variaion in he speaer-independen models MLLR adapaion of variances no very effecive Soluion: Speaer Adapive Training Adap (MLLR each raining speaer o he speaer-independen model. Use he adaped raining daa for each speaer o rain a new speaer-independen model Reduces he variaion by moving all speaers owards heir common average. Requires adapaion during recogniion March 29-30, 2007 Speech recogniion course 2007 Mas Blomberg 39

40 MLLR performance Models Relaive Error Reducion CHMM Baseline MLLR on mean only +2% MLLR on mean and +2% variance MLLR SAT +8% One conex-independen regression class for all conexdependen phones wih same mid uni March 29-30, 2007 Speech recogniion course 2007 Mas Blomberg 40

41 9.6.3 MLLR and MAP Comparison MLLR beer for small adapaion daa, MAP is beer when he adapaion daa is large. Combined MLLR+MAP bes in boh cases March 29-30, 2007 Speech recogniion course 2007 Mas Blomberg 4

42 9.6.4 Clusered Models A single speaer- and environmen- independen model ofen has oo much variabiliy for high performance recogniion and adapaion Cluser he raining daa ino smaller pariions Gender-dependen models can reduce WER by 0% Speaer clusering can reduce i furher, bu no as much Environmen clusering and adapaion in Chaper 0 March 29-30, 2007 Speech recogniion course 2007 Mas Blomberg 42

Speaker Adaptation Techniques For Continuous Speech Using Medium and Small Adaptation Data Sets. Constantinos Boulis

Speaker Adaptation Techniques For Continuous Speech Using Medium and Small Adaptation Data Sets. Constantinos Boulis Speaker Adapaion Techniques For Coninuous Speech Using Medium and Small Adapaion Daa Ses Consaninos Boulis Ouline of he Presenaion Inroducion o he speaker adapaion problem Maximum Likelihood Sochasic Transformaions