New Statistical Model for the Enhancement of Noisy Speech

Transcription

1 New Statistical Model for the Enhancement of Noisy Speech Electrical Engineering Department Technion - Israel Institute of Technology February 22, 27

2 Outline Problem Formulation and Motivation 1 Problem Formulation and Motivation Spectral Enhancement Speech Models Objectives 2 Spectral Analysis GARCH Model Speech Modeling 3 Variance Estimation Relation to Decision-Directed Estimation Experimental Results 4

3 Spectral Enhancement Spectral Enhancement Speech Models Objectives Let {Y tk } denote a noisy speech signal in the STFT domain: H1 tk Y tk = X tk + D tk H tk Y tk = D tk. The spectral enhancement problem can be formulated as { ( min E d X tk, ˆX ) tk ˆptk, ˆλ tk, σ } tk 2, Y tk ˆX tk ) d (X tk, ˆX tk - distortion measure between X tk and ˆX tk ˆp tk = P ( H1 tk ψ t) - speech presence probability estimate ˆλ tk = E { X tk 2 H1 tk, ψ t} - speech spectral variance estimate σ tk 2 = E { Y tk 2 H tk, ψ t} - noise spectral variance estimate ψ t - information employed for estimation at frame t (e.g., noisy data observed through time t)

4 Spectral Enhancement (cont.) Spectral Enhancement Speech Models Objectives In particular, assuming a squared error distortion measure of the form ) d (X tk, ˆX tk = g(ˆx tk ) g(x tk ) 2 where g(x) and g(x) are specific functions of X (e.g., X, X, log X, e j X ) the estimator ˆX tk is calculated from { g(ˆx tk ) = E g(x tk ) ˆp tk, ˆλ tk, σ } tk 2, Y tk { = ˆp tk E g(x tk ) H1 tk, ˆλ tk, σ } tk 2, Y tk { } +(1 ˆp tk )E g(x tk ) H tk, Y tk.

5 Spectral Enhancement (cont.) Spectral Enhancement Speech Models Objectives The design of a particular estimator for X tk requires the following specifications: Functions g(x) and g(x), which determine the fidelity criterion of the estimator. A conditional probability density function (pdf) p ( X tk λ tk, H1 tk ) for Xtk under H1 tk given its variance λ tk, which determines the statistical model. An estimator ˆλ tk for the speech spectral variance. An estimator σ tk 2 for the noise spectral variance. An estimator ˆp tk t 1 = P ( H1 tk ψ t 1) for the a priori speech presence probability, where ψ t 1 represents the information set known prior to having the measurement Y tk.

6 Speech Models Problem Formulation and Motivation Spectral Enhancement Speech Models Objectives Given {λ tk } and the state of speech presence in each time-frequency bin (H1 tk or H tk ), the speech spectral coefficients {X tk } are generated by X tk = λ tk V tk where { V tk H tk } { are identically zero, and Vtk H1 tk } are statistically independent complex random variables with zero mean, unit variance, and iid real and imaginary parts: H tk H tk 1 : E {V tk} =, E { V tk 2} = 1 : V tk =

7 Speech Models (cont.) Spectral Enhancement Speech Models Objectives Gaussian model [McAulay and Malpass, 198; Ephraim and Malah, 1984] ( ) p V ρtk H1 tk = 1 exp ( V 2 ) π ρtk ρ {R,I },V Rtk R {V tk },V Itk I {V tk } p(v) v

8 Speech Models (cont.) Spectral Enhancement Speech Models Objectives Gaussian model [McAulay and Malpass, 198; Ephraim and Malah, 1984] ( ) p V ρtk H1 tk = 1 exp ( V 2 ) π ρtk ρ {R,I },V Rtk R {V tk },V Itk I {V tk } Gamma model [Porter and Boll, 1984; Martin, 22] ( p ) V ρtk H1 tk = 1 2 π ( ) ( 3 1/4 V ρtk 1/2 exp 2 ) 3 2 V ρtk Laplacian model [Martin and Breithaupt, 23; Lotter and Vary, 23] ( ) p V ρtk H1 tk = exp ( 2 V ρtk ).

9 Speech Models (cont.) Spectral Enhancement Speech Models Objectives Gaussian model [McAulay and Malpass, 198; Ephraim and Malah, 1984] ( ) p V ρtk H1 tk = 1 exp ( V 2 ) π ρtk ρ {R,I },V Rtk R {V tk },V Itk I {V tk } Gaussian Gamma Laplacian p(v) v

10 Speech Models (cont.) Spectral Enhancement Speech Models Objectives Over the past two decades, the decision-directed approach has become the acceptable estimation method for variances of speech spectral coefficients [Ephraim and Malah, 1984] { ˆλ tk = max α ˆX t 1,k 2 + (1 α) ( } Y tk 2 σtk) 2, ξmin σtk 2.

11 Speech Models (cont.) Spectral Enhancement Speech Models Objectives Over the past two decades, the decision-directed approach has become the acceptable estimation method for variances of speech spectral coefficients [Ephraim and Malah, 1984] { ˆλ tk = max α ˆX t 1,k 2 + (1 α) ( } Y tk 2 σtk) 2, ξmin σtk 2. The decision-directed approach is not supported by a statistical model. α and ξ min have to be determined by simulations and subjective listening tests for each particular setup of time-frequency transformation and speech enhancement algorithm. α and ξ min are not adapted to the speech components.

12 Objectives Problem Formulation and Motivation Spectral Enhancement Speech Models Objectives Analyze the time-frequency correlation of speech and noise signals in the STFT domain. Formulate statistical models for speech signals in the STFT domain, which take into consideration the time-frequency correlation and heavy-tailed distribution of the expansion coefficients. Derive estimators for the speech spectral variances, which are based on the proposed models. Show that a special case of the proposed variance estimator degenerates to a decision-directed estimator.

13 Spectral Analysis Spectral Analysis GARCH Model Speech Modeling Clean speech signals, 16 khz, STFT using Hamming windows, 512 samples length (32 ms), 256 samples framing step (5% overlap). Scatter plots for successive spectral magnitudes: White Gaussian noise Speech, k = 17 (5Hz) 2 4 Dt+1,k (db) D tk (db) Xt+1,k (db) X tk (db)

14 Spectral Analysis (cont.) Spectral Analysis GARCH Model Speech Modeling Sample autocorrelation coefficient sequences (ACSs) along time-trajectories: Magnitude, 5Hz, 5% overlap Phase, 5Hz, 5% overlap Sample Autocorrelation 1.5 Sample Autocorrelation 1.5 Sample Autocorrelation Lag (frames) Magnitude, 2kHz, 5% overlap Lag (frames) Lag (frames) Magnitude, 5Hz, 75% overlap Sample Autocorrelation Lag (frames)

15 Spectral Analysis (cont.) Spectral Analysis GARCH Model Speech Modeling Typical variation of ρ(1), the correlation coefficient between successive spectral magnitudes: 1.8 Variation on frequency ρ(1) Frequency (khz) Correlation Coefficient Overlap Between Frames (%) Variation on overlap 1kHz (solid) 2kHz (dashed) noise (dotted)

16 Spectral Analysis (cont.) Spectral Analysis GARCH Model Speech Modeling When observing a time series of successive expansion coefficients in a fixed frequency bin, successive magnitudes of the expansion coefficients are highly correlated, whereas successive phases are nearly uncorrelated. Hence, the expansion coefficients are clustered in the sense that large magnitudes tend to follow large magnitudes and small magnitudes tend to follow small magnitudes, while the phase is unpredictable. Speech signals in the STFT domain are characterized by volatility clustering and heavy-tailed distribution.

17 Spectral Analysis GARCH Model Speech Modeling Generalized Autoregressive Conditional Heteroscedasticity (GARCH) Model GARCH models [Engle, 1982; Bollerslev, 1986] are widely used in various financial applications such as risk management, option pricing, and foreign exchange. They explicitly parameterize the time-varying volatility in terms of past conditional variances and past squared innovations (prediction errors), while taking into account excess kurtosis (i.e., heavy tail behavior) and volatility clustering, two important characteristics of financial time-series. Modeling speech expansion coefficients as GARCH processes offers a reasonable model on which to base the variance estimation, while taking into consideration the heavy-tailed distribution.

18 GARCH Model (cont.) Spectral Analysis GARCH Model Speech Modeling 4.2 Deutschmark/British Pound Foreign Exchange Rate Exchange Rate Jan 1984 Jan 1986 Jan 1988 Jan Deutschmark/British Pound Daily Returns Return Jan 1984 Jan 1986 Jan 1988 Jan

19 GARCH Model (cont.) Spectral Analysis GARCH Model Speech Modeling Let {y t } denote a real-valued discrete-time stochastic process, and let ψ t denote the information set available at time t. The innovation process in the MMSE sense is given by ε t = y t E {y t ψ t 1 } and the conditional variance (volatility) of y t is defined as σ 2 t = var {y t ψ t 1 } = E { ε 2 t ψ t 1 }. A GARCH model of order (p,q), denoted by ε t GARCH(p,q), has the following general form ε t σ 2 t = σ t z t = f ( σt 1 2,..., σ2 t p, ε2 t 1..., ) ε2 t q where {z t } is a zero-mean unit-variance white noise process with some specified probability distribution.

20 GARCH Model (cont.) Spectral Analysis GARCH Model Speech Modeling The widely used GARCH model assumes a linear formulation, q p σt 2 = κ + α i ε 2 t i + β j σt j 2, (1) i=1 j=1 and the values of the parameters are constrained by κ >, α i, β j, i = 1,...,q, j = 1,...,p, (sufficient constraints to ensure that the conditional variances {σt 2} are strictly positive) and by q p α i + β j < 1 i=1 (necessary and sufficient constraint for the existence of a finite unconditional variance of the innovations process). j=1

21 Speech Modeling Spectral Analysis GARCH Model Speech Modeling As before, given {λ tk } and the state of speech presence in each time-frequency bin (H1 tk or H tk), {X tk} are generated by X tk = λ tk V tk where { V tk H1 tk } are statistically independent complex random variables H tk H tk 1 : E {V tk} =, E { V tk 2} = 1 : V tk = However, {λ tk } are hidden from direct observation even under perfect conditions of zero noise (D tk = for all tk).

22 Speech Modeling (cont.) Spectral Analysis GARCH Model Speech Modeling Our approach is to assume that {λ tk } themselves are random variables, and to introduce conditional variances which are estimated from the available information. Let λ tk τ E { X tk 2 H1 tk, X τ } denote the conditional variance of X tk under H1 tk given the clean spectral coefficients up to frame τ. We assume that λ tk t 1, referred to as the one-frame-ahead conditional variance, is a random process which evolves as a GARCH(1,1) process: λ tk t 1 = λ min + µ X t 1,k 2 + δ ( ) λ t 1,k t 2 λ min where λ min >, µ, δ, µ + δ < 1 are the standard constraints imposed on the parameters of the GARCH model.

23 Variance Estimation Variance Estimation Relation to Decision-Directed Estimation Experimental Results Following the rational of Kalman filtering: Start with an estimate ˆλ tk t 1, and update the variance by using the additional information Y tk, Update step: } ˆλ tk t = E { X tk 2 ˆλtk t 1,Y tk Propagate the variance estimate ahead in time to obtain a conditional variance estimate at frame t + 1, Propagation step: ) ˆλ t+1,k t = λ min + µ ˆλ tk t + δ (ˆλ tk t 1 λ min The propagation and update steps are iterated, to recursively estimate the speech variances as new data arrive.

24 Variance Estimation Relation to Decision-Directed Estimation Experimental Results Relation to Decision-Directed Estimation For a Gaussian-GARCH model, the update step can be written as ( ) ˆλ tk t = α tk ˆλtk t 1 + (1 α tk ) Y tk 2 σtk 2 with α tk 1 ˆλ 2 tk t 1 (ˆλtk t 1 + σ 2 tk) 2. Using the propagation step with µ 1 and applying the lower bound constraint to ˆλ tk t rather than ˆλ tk t 1, we have { ˆλ tk t = max α tk ˆλt 1,k t 1 + (1 α tk ) ( Y tk 2 σtk 2 ) }, λmin

25 Variance Estimation Relation to Decision-Directed Estimation Experimental Results Relation to Decision-Directed Estimation (cont.) Recall the heuristically motivated decision-directed estimator [Ephraim and Malah, 1984] ˆλ tk = max { α ˆX t 1,k 2 + (1 α) ( } Y tk 2 σtk) 2, ξmin σtk 2 A special case of the GARCH-based variance estimator degenerates to a decision-directed estimator with a time-varying frequency-dependent weighting factor α tk α α tk ξ min σ 2 tk λ min ˆX t 1,k 2 ˆλ } t 1,k t 1 E { X t 1,k 2 ˆλt 1,k t 2,Y t 1,k

26 Experimental Results Variance Estimation Relation to Decision-Directed Estimation Experimental Results Speech signals: 2 different utterances from 2 different speakers, sampled at 16 khz and degraded by white Gaussian noise with SNRs in the range [,2]dB. Eight different speech enhancement algorithms are compared Algorithm Statistical Variance Fidelity # Model Estimation Criterion 1 Gaussian GARCH MMSE 2 Gamma GARCH MMSE 3 Laplacian GARCH MMSE 4 Gaussian Decision-Directed MMSE 5 Gamma Decision-Directed MMSE 6 Laplacian Decision-Directed MMSE 7 Gaussian GARCH MMSE-LSA 8 Gaussian Decision-Directed MMSE-LSA

27 Experimental Results (cont.) Variance Estimation Relation to Decision-Directed Estimation Experimental Results 8 Clean speech signal 8 Noisy signal, SNR = 5dB LSD = 13.75dB, PESQ= 1.76 Frequency [khz] Frequency [khz] Amplitude Time [Sec] Amplitude Time [Sec] Decision-Directed, MMSE-LSA GARCH, MMSE-LSA LSD = 9.dB, PESQ= 2.57 LSD = 3.59dB, PESQ= Frequency [khz] Frequency [khz] Amplitude Time [Sec] Amplitude Time [Sec]

28 Experimental Results (cont.) Variance Estimation Relation to Decision-Directed Estimation Experimental Results Log-Spectral Distortion (LSD) Input GARCH modeling method Decision-Directed method SNR Gaussian Gamma Laplacian Gaussian Gamma Laplacian [db] MMSE LSA MMSE MMSE MMSE LSA MMSE MMSE Perceptual Evaluation of Speech Quality (PESQ) scores (ITU-T P.862) Input GARCH modeling method Decision-Directed method SNR Gaussian Gamma Laplacian Gaussian Gamma Laplacian [db] MMSE LSA MMSE MMSE MMSE LSA MMSE MMSE

29 Experimental Results (cont.) Variance Estimation Relation to Decision-Directed Estimation Experimental Results The GARCH modeling method yields lower LSD and higher PESQ scores than the decision-directed method. Using the decision-directed method, a Gaussian model is inferior to Gamma and Laplacian models. Using the GARCH modeling method, a Gaussian model is superior to Gamma and Laplacian models. It is difficult, or even impossible, to derive analytical expressions for MMSE-LSA estimators under Gamma or Laplacian models. The GARCH modeling method facilitates MMSE-LSA estimation, while taking into consideration the heavy-tailed distribution.

30 The decision-directed approach is heuristically motivated, and cannot be adapted to the components of the speech signal. Speech signals in the STFT domain demonstrate both volatility clustering and heavy tail behavior. GARCH modeling provides a new framework for optimal restoration of speech signals in adverse environments. The concepts of GARCH and hidden Markov models can be used to develop Markov-Switching GARCH models for multichannel speech processing, voice activity detection, source localization, dereverberation and adaptive beamforming in noisy and reverberant environments.