Graphical Models for Automatic Speech Recognition

Transcription

1 Graphical Models for Automatic Speech Recognition Advanced Signal Processing SE 2, SS05 Stefan Petrik Signal Processing and Speech Communication Laboratory Graz University of Technology GMs for Automatic Speech Recognition p. 1

2 Introduction to ASR Pronunciation Modeling Language Modeling Basic Speech Models Advanced Speech Models Summary Overview GMs for Automatic Speech Recognition p. 2

3 Introduction to ASR Find most likely sequence of words w, given observations X w = arg max(p(w X)) = arg max w w P(w) P(X w) P(X) w = w 1...w m : sequence of words X : feature vectors P(w) : language model P(X w) : acoustic model Tasks in ASR: acoustic modeling pronunciation modeling language modeling GMs for Automatic Speech Recognition p. 3

4 Pronunciation Modeling Map base-forms (word dictionary based pronunciations) to surface forms (actual pronunciations) Use 1 st order Markov chain for representation Phones are shared across multiple words: /b/ae/g/ /b/ae/t/ Solution 1: Expanded model increase state space of Q t, to model not only phone but also position in word condition on word W t and sequential position of phone S t : P(Q t W t,s t ) GMs for Automatic Speech Recognition p. 4

5 Pronunciation Modeling Solution 2: Parameter-tied model avoids expanded state space by parameter tying and sequence control S t + 1 if R t = 1 p(s t+1 = i R t,s t ) = δ i,f(rt,s t ) S t+1 = else S t GMs for Automatic Speech Recognition p. 5

6 Pronunciation Modeling Solution 3: Decision trees input node I, output node O, decision RVs R i P(D l = i I) = δ i,fl (I,d 1:l 1 ) with decisions d l = f l (I,d 1:l 1 ) GMs for Automatic Speech Recognition p. 6

7 Language Modeling Predict next word from history of previous words Joint distribution: p(w 1:T ) = n p(w t W 1:t 1 ) t=1 Restrict to history of last n 1 words: p(w t W t n+1:t 1 ) = p(w t H t ) Problem: sparse data Solution: smoothing p(w t w t 1,w t 2 ) = α 3 (w t 1,w t 2 )f(w t w t 1,w t 2 ) + α 2 (w t 1,w t 2 )f(w t w t 1 ) + α 1 (w t 1,w t 2 )f(w t ) GMs for Automatic Speech Recognition p. 7

8 n-grams Switching parents: value-specific conditional independence P(C M1,F1,M2,F2) = i P(C M i,f i,s = i)p(s R i ) GMs for Automatic Speech Recognition p. 8

9 Resulting model: n-grams p(w t w t 1,w t 2 ) = α 3 (w t 1,w t 2 )f(w t w t 1,w t 2 ) + α 2 (w t 1,w t 2 )f(w t w t 1 ) + α 1 (w t 1,w t 2 )f(w t ) GMs for Automatic Speech Recognition p. 9

10 Class-Based Language Model Idea: cluster words together and form a Markov chain over the groups Much lower dimensional class variables C i than high-dimensional word variables W i Syntactic, semantic or pragmatic grouping: parts-of-speech: nouns, verbs, adjectives, determiners,... numerals, colors, sizes, physical values,... animals, plants, vegetables, people,... GMs for Automatic Speech Recognition p. 10

11 Class-Based Language Model Introduce token unk with non-zero probability for unknown words Vocabulary W = {unk} S M with p ml (w W) = N(w) N Constraint: p(unk) = 0.5 p ml (S) = 0.5 w S p ml(w) Resulting probability model: p d (w) = 0.5p ml (S) if w = unk 0.5p ml (w) if w S p ml (w) otherwise Condition on current class: p d (w c) GMs for Automatic Speech Recognition p. 11

12 Class-Based Language Model Graphical model: Additional observed variable V t which is always V t = 1 K t,b t : switching parents C t : word class W t : word Show p(w t,v t = 1 c t ) = p d (w t c t ) GMs for Automatic Speech Recognition p. 12

13 Class-Based Language Model Conditional distributions: p(k t c t ) = p(s c t ) if k t = 1 1 p(s c t ) otherwise p(b t = 0) = p(b t = 1) = 0.5 p M (w c) = p m l(w c) p(m c) if w M 0 otherwise p S (w c) = p m l(w c) p(s c) if w S 0 otherwise p(w t k t,b t,c t ) = p M (w t c t ) if k t = 0 p S (w t c t ) if k t = 1 and b t = 1 δ {wt =unk} if k t = 1 and b t = 0 GMs for Automatic Speech Recognition p. 13

14 Basic Speech Models Hidden Markov Model (HMM): encompasses acoustic, pronunciation, and language modeling hidden chain corresponds to seq. of words, phones and sub-phones hidden states Q 1:T and observations X 1:T Q t:t Q 1:t 2 Q t 1 and X t {Q t,x t } Q t either DGM or UGM: moralizing the graph introduces no new edges and result is already triangulated GMs for Automatic Speech Recognition p. 14

15 Basic Speech Models HMMs with mixture-of-gaussians output: explicit modeling of mixture variable p(x t Q t = q,c t = i) = N(x t ;µ q,i,σ q,i ) p(c t = i Q t = q) = C(q,i) Semi-continuous HMMs: single, global pool of Gaussians, each state corresponds to a particular mixture over the pool p(x Q = q) = i p(c = i Q = q)p(x C = i) GMs for Automatic Speech Recognition p. 15

16 Basic Speech Models Auto-regressive HMMs (AR-HMM): relaxes conditional independence constraint 2: X t 1 helps predicting X t result: models with higher likelihood note: not to be confused with linear predicitve HMMs GMs for Automatic Speech Recognition p. 16

17 Basic Speech Models Input/Output HMMs (IOHMM): variables corresponding to input and output at each time frame given input feature stream X 1:T, try to find E[Y X] CPD for Q t as 3-dim array: P(Q t = j Q t 1 = i,x t = k) = A(i,j,k) promising for speech enhancement GMs for Automatic Speech Recognition p. 17

18 Advanced Speech Models Factorial HMMs: distributed representation of the hidden state special case HMM with tied parameters and state transition restrictions conversion to HMM possible, but inefficient: complexity changes from O(TMK M+1 ) to O(TK 2M ) GMs for Automatic Speech Recognition p. 18

19 Advanced Speech Models Mixed-memory HMMs: like factorial HMM, but fewer parameters (two 2-dimensional tables instead of single 3-dimensional one) cond. independence: Q t R t 1 S t = 0 and Q t Q t 1 S t = 1 p(q t Q t 1,R t 1 ) = p(q t Q t 1,S t = 0)P(S t = 0) + p(q t Q t 1,S t = 1)P(S t = 1) GMs for Automatic Speech Recognition p. 19

20 Segment models: Advanced Speech Models each HMM state can generate sequence of observations, not just single one overall joint distribution: p(x 1:T = x 1:T ) = τ p(x t(i,1) ), p(x t(i,2) ),..., p(x t(i,li )), l i q i, τ)p(q i q i 1, τ)p(τ) τ q 1:τ l 1:τ i=1 observation segment distribution: p(x 1, x 2,..., x l, l q) = p(x 1, x 2,..., x l l, q)p(l q) plain HMM if p(x 1, x 2,..., x l l, q) = l j=1 p(x j q) and p(l q) geometric dist. GMs for Automatic Speech Recognition p. 20

21 Advanced Speech Models Buried Markov Model (BMM): HMM s cond. independence structure may not accurately model data additional edges between observation vectors needed Idea: measure contextual information of a hidden variable conditional mutual information: additional information X <t provides about X t not already provided by Q t I(X t ;X <t Q t ) = > 0 add edge I(X t ;X <t Q t = q)p(q t = q) = q 0 no change underlying Markov chain in HMM is further hidden (buried) by specific cross-observation dependencies GMs for Automatic Speech Recognition p. 21

22 Advanced Speech Models Buried Markov Model (BMM): for learning, measure discriminative mutual information between X and its potential set of parents Z EAR (explaining away residual): EAR(X,Z) = I(X;Z Q) I(X;Z) arg max Z EAR(X, Z) optimized posterior probability for Q GMs for Automatic Speech Recognition p. 22

23 Summary Some well-known speech models presented in terms of graphical models Used for acoustic, pronunciation and language modeling Standard HMM approach can be improved by GMs with relaxed conditional independence statements More models available... GMs for Automatic Speech Recognition p. 23

24 References Jeffrey A. Bilmes, Graphical Models and Automatic Speech Recognition, 2003 Kevin Patrick Murphy, Dynamic Bayesian Networks: Representation, Inference and Learning, 2002 Jeffrey A. Bilmes, Dynamic Bayesian Multinets, 2000 Jeffrey A. Bilmes, Data-Driven Extensions to HMM Statistical Dependencies, 1998 GMTK: The Graphical Models Toolkit, GMs for Automatic Speech Recognition p. 24