T Automatic Speech Recognition: From Theory to Practice

Transcription

1 Auomaic Speech Recogniion: From Theory o Pracice hp:// November 15, 2004 Prof. Bryan Pellom Deparmen of Compuer Science Cener for Spoken Language Research Universiy of Colorado pellom@cslr.colorado.edu Auomaic Speech Recogniion: From Theory o Pracice 1

2 Announcemens I sill need 2 more voluneers o presen heir projec opic (nex week) on November 22 nd The goal is o presen o he class (and myself) your chosen opic area. Brief 10 minue presenaion (projec overview) Does no have o reflec your compleed projec (since ha is due December 8 h ). Auomaic Speech Recogniion: From Theory o Pracice 2

3 Today How o reduce mismach beween raining and es condiions o improve recogniion reliabiliy Noise Robusness Techniques Speaker Adapaion Techniques Auomaic Speech Recogniion: From Theory o Pracice 3

4 Classic References Boll, S. F. (1979) Suppression of Acousic Noise in Speech Using Specral Subracion", IEEE Transacions on Acousics, Speech, and Signal Processing, Vol. ASSP-27, pp , April. Ephraim, Y. and Malah, D. (1984) "Speech Enhancemen Using a Minimum Mean- Square Error Shor-Time Specral Ampliude Esimaor", IEEE Trans. ASSP, Vol. ASSP-32, No. 6, pp Hansen, J. H.L. and Clemens, M. A. (1991) "Consrained Ieraive Speech Enhancemen Wih Applicaion o Speech Recogniion", IEEE Transacions on Signal Processing, Vol. 39, No. 4, pp , April. Gong, Y. (1995) "Speech recogniion in noisy environmens: A survey", Speech Comm., Vol. 16, pp Gales, M. J. F. (1997) Maximum Likelihood Linear Transformaions for HMM-Based Speech Recogniion, Technical Repor CUED/F-INFENG/TR 291, Cambridge Universiy, May. Huang, X., Acero, A., Hon, H.-W., (2001) Spoken Language Processing: A Guide o Theory, Algorihm, and Sysem Developmen, Prenice Hall, ISBN Auomaic Speech Recogniion: From Theory o Pracice 4

5 Training vs. Tes Mismach Performance of speech recogniion sysems degrades whenever here is a mismach beween raining and es condiions Mismach can occur a various levels of processing and o various degrees, Acousic Modeling Language Modeling Pronunciaion Modeling Auomaic Speech Recogniion: From Theory o Pracice 5

6 Acousic Variabiliy Environmenal Transducer Channel & Codec Noise Speaker Accen Dialec Vocal Trac Geomery Lombard Effec Age & Gender Microphone frequency response Telephone band-limiing, VoIP Addiive, Impulsive, ec. Foreign Accen Regional Differences Voice changes due o noise Auomaic Speech Recogniion: From Theory o Pracice 6

7 Language Variabiliy Speaker-Specific Choice of words is highly speaker-specific Vocabulary varies speaker-o-speaker Task-Specific Topic shifs impac word choices, vocabulary, and saisical disribuions of words We can no expec a language model rained on financial news o work well when ranscribing spors news. Auomaic Speech Recogniion: From Theory o Pracice 7

8 Impac of Addiive Noise on Word Error Rae Train on noise-free daa, es on noisy daa Auomaic Speech Recogniion: From Theory o Pracice 8

9 Muli-Syle Training Train on acousic daa ha has been corruped by (1) differen noise ypes, and (2) differen noise levels. Works well when all ypes of operaing environmens are known. Can always predic he environmen; Need mehods o compensae for noise + channel Auomaic Speech Recogniion: From Theory o Pracice 9

10 Muli-Syle Training Train on daa corruped by noise of various levels. Auomaic Speech Recogniion: From Theory o Pracice 10

11 Robus Acousic Modeling Robus Fron-end Processing Remove noise from speech Design feaures o be as noise, channel robus as possible Feaure Compensaion Reduce he observed mismach beween he exraced feaures and esimaed model parameers Acousic Model Compensaion Modify acousic model parameers o closer mach he observed es environmen Auomaic Speech Recogniion: From Theory o Pracice 11

12 Robus Acousic Modeling Audio Noise Suppression Noise Robus Feaure Exracion Feaure Adapaion Acousic Model Model Adapaion Search Auomaic Speech Recogniion: From Theory o Pracice 12

13 Fron-End Noise Suppression Auomaic Speech Recogniion: From Theory o Pracice 13

14 Scope of he Problem (A) (B) Error Rae for WSJ Dicaion sysem rained on clean speech (Paul & Baker, 1992) (A) Word Error Rae as a funcion of SNR for Whie Noise (B) Word Error Rae as a funcion of Microphone Disance Auomaic Speech Recogniion: From Theory o Pracice 14

15 Microphone Arrays Microphone arrays provide spaial seleciviy Spaial seleciviy varies as a funcion of frequency Example: Filer-and-sum beam former, x 1[ n ] x 2[ n ] x N [n] τ 1 τ 2 τ N H 1( z ) H 2 ( z ) H N (z) Σ y[n] y[ n] = M 1 P 1 m= 0 p= 0 h m [ p] x Auomaic Speech Recogniion: From Theory o Pracice 15 m [ n p τ m ]

16 Noise Suppression Goal is o reduce he impac of noise on speech Typical approaches esimae he clean-speech magniude specrum from he noisy-speech magniude specrum Phase of he noisy signal used as he esimae of he phase of he clean-speech signal Auomaic Speech Recogniion: From Theory o Pracice 16

17 Saionary Addiive Noise Relaively consan specral shape across ime Has a non-uniform impac on speech, Speech sounds have differen specral shapes across ime Speech sounds have differen energy levels across ime Signal-o-noise raio herefore is a funcion of ime even when he noise is addiive and saionary. Auomaic Speech Recogniion: From Theory o Pracice 17

18 Original 5dB SNR Car Highway Noise 5dB SNR Whie Noise Auomaic Speech Recogniion: From Theory o Pracice 18

19 Auomaic Speech Recogniion: From Theory o Pracice 19 Addiive Noise Model ) ( ) ( ) ( n d n s n y + = ) ( ) ( ) ( ω ω ω D S Y + { } ( ) ( ) ω ω ω D S n d n s F Y + = + = ) ( ) ( ) ( ( ) ( ) ( ) ( ) ( ) ( ) 2Re{ } ) ( * ω ω ω ω ω ω ω D S D S D S Y + + = + =

20 Specral Subracion Subrac esimae of noise power specrum from observed (speech+noise) power specrum { } D ( ω) ˆ 2 2 ( ω) ( ω) E 2 = Y S Mus ensure ha specral esimae is posiively valued. Compue feaures from esimaed clean speech power specrum Auomaic Speech Recogniion: From Theory o Pracice 20

21 (a) Specral Subracion Example (b) (c) (a) Original clean, (b) degraded 10dB WGN, (c) Enhanced Auomaic Speech Recogniion: From Theory o Pracice 21

22 Improvemens from Specral Subracion ** Many imes, hese gains are no as significan in realworld operaing environmens. Someimes specral subracion can degrade ASR performance! Auomaic Speech Recogniion: From Theory o Pracice 22

23 Wiener Filering Esimae opimal Weiner filer and apply o he noisy speech specral magniudes, ˆ ( ω ) = Y ( ω ) H ( ω ) H ( ω ) S k k k k = S 2 S ( ωk ) ( ω ) 2 + D ( ω ) 2 k k Numeraor for filer (H) is unknown, mus be esimaed. Someimes his is done ieraively using LPC-based models for speech as a consrain in he esimaion process. J. H. L. Hansen, M. A. Clemens, "Consrained Ieraive Speech Enhancemen Wih Applicaion o Speech Recogniion", IEEE Transacions on Signal Processing, Vol. 39, No. 4, pp , April, Auomaic Speech Recogniion: From Theory o Pracice 23

24 Minimum Mean-Square Error (MMSE) Specral Ampliude Esimaion Esimae he clean speech specral ampliude from corruped speech using MMSE mehods Sˆ Y. Ephraim and D. Malah, Speech Enhancemen Using Minimum Mean Square Error Shor-Time Specral Ampliude Esimaor, IEEE Trans. On Acousics, Speech, and Signal Processing, Vol. 32, No. 6, pp , 1984 Derived MMSE Esimaor: ( ω ) = Y ( ω ) H ( ω ) k k ξk γ k + ξ υ ( ) k υ k υ k v k π k H ωk = exp ( 1+ υ k ) I o + υ k I1 2 γ k Auomaic Speech Recogniion: From Theory o Pracice 24 k υ = k ξ k γ 1 k :a priori SNR :a poseriori SNR

25 Several Issues wih Speech Enhancemen Approaches Requires esimae of noise o be updaed during periods of silence (noisy-only regions). How o do his well? Mos algorihms subrac variable amouns of he noise esimae o obain rade-offs in disorion vs. noise aenuaion. How does his impac he ASR engine? Algorihms developed o improve inelligibiliy may no necessarily improve ASR accuracy. Someimes worse! Many speech enhancemen algorihms have no been formulaed o improve ASR accuracy. Be careful! Auomaic Speech Recogniion: From Theory o Pracice 25

26 Considering he Communicaion Channel Communicaion channels generally ac o filer he inpu signal (muliplicaive disorion in he frequency domain) Analog elephone neworks band-limi he signal o a range of approximaely 200Hz o 3400Hz. Specral shape of channel can vary from call-o-call, someimes wih echo. Oher ypes of channels, Voice over IP Variabiliy due o Microphones Telephone handse variabiliy (also wireless phones) Cellular elephony Auomaic Speech Recogniion: From Theory o Pracice 26

27 Noisy-Channel Model s(n) h(n) y(n) d(n) y ( n) = s( n) h( n) + d( n) s(n): clean speech signal h(n): channel (elephone, microphone) d(n): addiive noise Auomaic Speech Recogniion: From Theory o Pracice 27

28 Auomaic Speech Recogniion: From Theory o Pracice 28 Specral Domain Noisy-Channel Model ) ( ) ( ) ( ) ( n d n h n s n y + = { } ) ( ) ( ) ( 2Re ) ( ) ( ) ( ) ( * ω ω ω ω ω ω ω D H S D H S Y + + = ) ( ) ( ) ( ) ( ω ω ω ω D H S Y + Assuming he speech and noise are saisically independen,

29 Robus Feaure Exracion Auomaic Speech Recogniion: From Theory o Pracice 29

30 MFCC Block Diagram Power specrum f (n) ( ) 2 DFT f ( n) Mel-Scale Filer Bank Energy from each filer e( j) j = 1LJ Log-Energy log(o) Discree Cosine Transform Compression & Decorrelaion Auomaic Speech Recogniion: From Theory o Pracice 30

31 Robus Feaure Exracion Aemps o compensae for channel and noise during feaure exracion process Recall ha he ypical cepsral parameer represenaion is based on he log-scaled oupus of a series of non-linear spaced filers Addiive Noise will impac he disribuions of he filerbank values, bu how? Auomaic Speech Recogniion: From Theory o Pracice 31

32 Impac of Noise on Mel-Scale Filer banks Auomaic Speech Recogniion: From Theory o Pracice 32

33 Classic Feaure Normalizaion Mehods Cepsral Mean Normalizaion (CMN) Mainly compensaes for channel (and some noise) Cepsral Variance Normalizaion Mainly compensaes for noise Vocal Trac Lengh Normalizaion (VTLN) Mainly compensaes for speaker-differences Auomaic Speech Recogniion: From Theory o Pracice 33

34 Impac of Noise on (Simulaed) Cepsral Parameers Assume noise and clean speech are Gaussianly disribued in log-specral domain, ( exp( s ) exp( d )) y = log + Assume speech wih mean=10, variance=5. Simulae he adding of noise a various levels Auomaic Speech Recogniion: From Theory o Pracice 34

35 Impac of Noise on (Simulaed) Cepsral Parameers Resuling disribuion iniially becomes bimodal, hen skewed unimodal Mean shifs Variance decreases wih increasing noise level Image from Liao and Gales (2004) Auomaic Speech Recogniion: From Theory o Pracice 35

36 Noisy-Channel Model in Log-Specral Domain Our channel model (k refers o filer bank bin), Y ( ω k ) = S ( ωk ) H ( ωk ) + D ( k ω ) 2 In he log-domain, log ( ) ( ) ( ) Y ( ωk ) = log S ( ωk ) + log H ( ωk ) + ( ( ( ) ( ) ( 2 ) exp log D ( ω ) log S ( ω ) log H ( ω )...log 1 k k k Auomaic Speech Recogniion: From Theory o Pracice 36

37 Noisy-Channel Model in Log-Specral Domain Define, g( a) = C log 1 ( ( 1 + exp C a ) Where C and C -1 are he discree cosine ransform (DCT) and inverse DCT. The cepsral parameers can hen be described by he following non-linear model, y ( c) = s ( d s h ) ( c) ( c) ( c) ( c) ( c) + h + g Auomaic Speech Recogniion: From Theory o Pracice 37

38 Cepsral Mean Normalizaion Channel disorion (h) in linear specral domain resul in addiive disorions in he log-specral (cepsral) domain y ( c) = s ( s h ) ( c) ( c) ( c) ( c) + h Telephone channel differences compensaed by subracing he long-erm mean from he cepsral feaures sˆ + Auomaic Speech Recogniion: From Theory o Pracice 38 g ( c) ( c) ( c) ( c) y = y yn ( c) 1 = y T T n= 1

39 Variaions on CMN CMN-2 Compue a running mean for he speech and silence separaely Deec speech vs. silence and use he appropriae mean Real-Time Implemenaions Running average (ypically 5 seconds): ( c) ( c) ( c) y = α y + ( 1 α) y 1 Auomaic Speech Recogniion: From Theory o Pracice 39

40 Cepsral Variance Normalizaion Addiive noise reduces he variance of cepsral feaures Compensae by normalizing all feaure componens o have variance of 1.0 Typically, compue sandard deviaion of feaures over large block of adapaion daa. Divide feaures by heir sandard deviaion Auomaic Speech Recogniion: From Theory o Pracice 40

41 Vocal Trac Lengh Normalizaion Vocal rac lenghs vary from speaker o speaker Influence forman frequency locaions Source of inra-speaker variabiliy VTLN mehods generally warp he speech specrum as o remove variabiliies due o vocal rac lengh Welling, L., Ney, H., and Kanahak, S. (2002) Speaker Adapive Modeling by Vocal Trac Normalizaion, IEEE Transacions on Speech and Audio Processing, Vol. 10, No. 6, pp , Sepember. Auomaic Speech Recogniion: From Theory o Pracice 41

42 Uniform Lossless Tube Model of Vocal Trac Glois Lips L 17.5 cm Forman locaions linearly relaed o ube lengh, c F = [2i 1] c = 350 m / i 4L i = 1,2,3,4,... sec Auomaic Speech Recogniion: From Theory o Pracice 42

43 VTLN via Linear Frequency Warping f ou f ou = α 0.88 α f in 1.12 f in Auomaic Speech Recogniion: From Theory o Pracice 43

44 MFCC wih VTLN Block Diagram Power specrum f (n) ( ) 2 DFT f ( n) Frequency Warp (α) Mel-Scale Filer Bank Energy from each filer e( j) j = 1LJ Log-Energy log(o) Discree Cosine Transform Compression & Decorrelaion Auomaic Speech Recogniion: From Theory o Pracice 44

45 VTLN Implemenaion Mus deermine frequency warp facor (alpha) for each speaker in raining se Apply warping during feaure exracion, rain acousic models During recogniion, mus deermine opimal frequency warp facor for each es speaker Auomaic Speech Recogniion: From Theory o Pracice 45

46 VTLN During HMM Training Consruc an iniial model λ using a single Gaussian mixure componen for each clusered riphone sae For each raining speaker, selec a frequency warping facor ha maximizes he likelihood of he raining daa given he reference ranscripion ( α ) α s = arg max P( O W, λ) α Esimae normalized acousic models by exracing feaures using speaker-dependen warp facors. Rerain HMMs using sandard decision ree mehod Auomaic Speech Recogniion: From Theory o Pracice 46

47 Speaker-Dependen Frequency Warp Facors Esimaed During Training Auomaic Speech Recogniion: From Theory o Pracice 47

48 VTLN During Recogniion Perform a firs pass recogniion using sandard acousic models (rained wih alpha=1.0). This provides an iniial esimae of he word srings. Selec a frequency warping facor ha maximizes he likelihood of he speaker s frequency warped feaures given he hypohesized ranscripion α s = arg max P( O Ŵ, λ) α Perform a second recogniion pass using VTLN normalized acousic models wih feaures exraced using he speaker-dependen frequency warp facor Auomaic Speech Recogniion: From Theory o Pracice 48 ( α )

49 Linear Discriminan Analysis (LDA) Improve Discriminaion via a linear ransform on feaures Maps a se of inpu feaure vecors (x) of dimension D o a se of oupu feaure vecors (y) of dimension D by means of a linear ransformaion θ, y ( c) = θ T x ( c) θ chosen o maximize he raio of beween-class scaer o wihin-class scaer given a se of class-labels on he raining se (x). Auomaic Speech Recogniion: From Theory o Pracice 49

50 Esimaion of LDA Transform Given a se of vecors assigned o a se of C classes, define Scaer-Marix as, S i = x c ( i) ( ( c) )( ( c) x m x m ) T Where m i is he mean of he ih class. i i Define Wihin-class Scaer, c S W = S i i= 1 Auomaic Speech Recogniion: From Theory o Pracice 50

51 Esimaion of LDA Transform Define Beween-Class Scaer, S B c = i=1 ( m m)( m m) T i i Can be shown ha he soluion for θ is he eigenvecors of he marix, S 1 S W B Auomaic Speech Recogniion: From Theory o Pracice 51

52 LDA as used in Speech Recogniion Vierbi Align raining daa using non-lda sysem (associae feaure vecors o phone or subphone unis) For 50 phones wih 3 saes per phone, we migh have 50x3 = 150 classes Take each raining vecor (jus saic 13-D cepsrum) and augmen i wih beween 4-8 surrounding vecors. Esimae an LDA ransform for his exended vecor represenaion. Keep he op N dimensions (based on eigen values) for recogniion (40-60). Rerain he acousic models wih his new feaure represenaion. Auomaic Speech Recogniion: From Theory o Pracice 52

53 Acousic Model Adapaion Auomaic Speech Recogniion: From Theory o Pracice 53

54 Acousic Model Adapaion For HMM saes modeled wih Gaussian disribuions, Model Adapaion Mehods aemp o shif he means and variances of Gaussians o beer mach he inpu feaure disribuions Example Techniques, Parallel Model Combinaion (PMC) Maximum Likelihood Linear Regression (MLLR) Maximum A Poseriori (MAP) Adapaion Auomaic Speech Recogniion: From Theory o Pracice 54

55 Parallel Model Combinaion (PMC) Acousic model compensaion for addiive and convoluional noise sources Assumes HMMs rained on clean speech Typically assumes noise is saionary and modeled using 1 HMM sae A clever approach ha modifies he (saic) cepsral means of modeled Gaussians Auomaic Speech Recogniion: From Theory o Pracice 55

56 Parallel Model Combinaion (PMC) Basic Idea, Conver each HMM cepsral mean vecors from log-specral domain o linear specral domain. Add noise specrum esimae o clean speech specrum from he acousic model Conver new noisy-speech specrum back o cepsral domain o ge compensaed HMM model mean vecors Auomaic Speech Recogniion: From Theory o Pracice 56

57 Parallel Model Combinaion (PMC) M.J.F. Gales and S.J. Young (1995). Robus Speech Recogniion in Addiive and Convoluional Noise using Parallel Model Combinaion. Compuer Speech and Language Volume 9. Auomaic Speech Recogniion: From Theory o Pracice 57

58 Maximum Likelihood Linear Regression (MLLR) Model adapaion echnique which esimaes new values for he Gaussian mean vecors via a linear ransform The linear ransform is esimaed from labeled raining daa (feaures and heir sae alignmens) M.J.F. Gales and P.C. Woodland (1996), Mean and Variance Adapaion wihin he MLLR Framework, Compuer Speech and Language Volume 10. Auomaic Speech Recogniion: From Theory o Pracice 58

59 Maximum Likelihood Linear Regression (MLLR) Esimaes a linear ransform marix (A) and bias vecor (b) o ransform HMM model means: µ new Transform esimaed o maximize he likelihood of he adapaion daa Speech sounds can be clusered ino a se of regression classes. Transform applied o all Gaussians wihin he same regression class. = A µ + r old Auomaic Speech Recogniion: From Theory o Pracice 59 b r

60 MLLR Variance Adapaion Can also adap he Gaussian variances under he MLLR framework (Gales & Woodland, 1997) Variance adapaion provides only minor benefi as a echnique for speaker adapaion 2-7%% relaive reducion in WER For speech recogniion in noisy environmens, adapaion of variances provides more benefi Assiss in compensaing for variance reducion due o noise Auomaic Speech Recogniion: From Theory o Pracice 60

61 Ieraive Unsupervised MLLR Typically MLLR is applied in an ieraive, and unsupervised manner, speech Recogniion word hypohesis, segmenaion MLLR HMMs Typical error reducions on he order of 15-20% relaive Auomaic Speech Recogniion: From Theory o Pracice 61

62 Word Error Rae vs. MLLR Ieraion Convergence is generally reached afer 2-5 ieraions of decoding followed by MLLR adapaion. Phoneme Recogniion (Ialian Children s Speech): Firs-Pass (WER) 18.1% MLLR Ier #1 16.0% MLLR Ier #2 15.3% MLLR Ier #3 14.9% MLLR Ier #4 14.7% MLLR Ier #5 14.6% 19.3% Relaive Error reducion Auomaic Speech Recogniion: From Theory o Pracice 62

63 Maximum A Poseriori Adapaion (MAP) MAP Adapaion can only be applied Gaussians ha are seen in he es daa, N$ α µ = m$ + new obs α α µ old N$ + N$ + Nˆ α mˆ obs Number of frames of adapaion daa Weigh for prior esimae of old mean Mean vecor of adapaion daa assigned o Gauss. Auomaic Speech Recogniion: From Theory o Pracice 63

64 MAP vs. MLLR MLLR Adapaion preferred over MAP for sparse adapaion ses MAP only re-esimaes seen daa componens Can esimae Block-Diagonal MLLR ransforms As adapaion daa increases, MAP generally ouperforms MLLR Why? MLLR based on a fixed se of regression classes Can combine MAP+MLLR Apply MLLR firs, Adap seen Gaussians wih MAP approach given sufficien daa Auomaic Speech Recogniion: From Theory o Pracice 64

65 Performance of MLLR and MAP Speaker independen MAP Speaker dependen MLLR MLLR+MAP Number of Adapaion Uerances Auomaic Speech Recogniion: From Theory o Pracice 65

66 Consrained MLLR (CMLLR) A variaion on MLLR Transform can be applied o he feaures raher han o he model Gaussians Typically boh CMLLR and MLLR are applied ogeher for added benefi Auomaic Speech Recogniion: From Theory o Pracice 66

67 Speaker Adapive Training (SAT) Aemps o remove speaker-specific characerisics from he raining daa o build robus speaker-independen models SAT using Feaure-Space CMLLR Transforms, Train sandard acousic models Esimae CMLLR feaure ransform for each raining speaker (use models from sep #1) Transform each speaker s feaures using speakerdependen CMLLR ransform Rerain acousic models Auomaic Speech Recogniion: From Theory o Pracice 67

68 Benchmarking Robus Sysems DARPA SPINE Speech Processing in Noisy Environmens (SPINE) Universiy and Commercial Several paricipans in Human-o-Human Dialogs in various noisy environmens 3,000 word vocabulary Word Error Raes ranging from 20-50% AURORA Popular in several of he las speech conferences End Goal: A sandard for Disribued Speech Recogniion Aemps o compare sysems using same daabase and recognizer, bu allow researchers o propose new fron-ends Auomaic Speech Recogniion: From Theory o Pracice 68

69 Wha o Expec from Adapaion Mos echniques are used in combinaion Typical Research Sysem migh use, CMN+CVN+LDA(HLDA)+VTLN+MLLR+CMLLR Speech Enhancemen fron-ends seem less common Typical Relaive Error Reducions Cepsral Mean Normalizaion ~ 5 % Cepsral Variance Normalizaion ~ 5 % LDA: 10 15% VTLN: 5 10% MLLR: (unsupervised) (supervised) 15 20% 25 35% Auomaic Speech Recogniion: From Theory o Pracice 69

70 Nex Week A few words abou hypohesis combinaion ASR Course Review for 30 minues Iniial (Shor) Projec Presenaions 10 minues presenaion abou your projec Auomaic Speech Recogniion: From Theory o Pracice 70