Timbral, Scale, Pitch modifications

Transcription

1 Introduction Timbral, Scale, Pitch modifications M2 Mathématiques / Vision / Apprentissage Audio signal analysis, indexing and transformation Page 1 / 40 Page 2 / 40 Modification of playback speed Modifications of duration and pitch Origin of the problem: y(t) = x(αt) Y (f ) = 1 α X( f α ) Goal: separately control the time and frequency scales Page 3 / 40 Page 4 / 40

2 Outline Separate control of the time and frequency scales Synthesis by means of wavetable sampling Post-synchronization of sound and video Musical post-production Three categories of methods: Spectral methods: phase vocoder Temporal methods: TD-PSOLA Parametric methods: LPC, sinusoids plus noise model Part I Definitions Page 5 / 40 Page 6 / 40 Vocal production model Signal models Time-varying, linear source / filter model: x(t) = + g(t, τ) e(t τ) dτ Frequency response of the filter: G(t, f ) = + g(t, τ) e j2πf τ dτ = M(t, f ) e jϕ(t,f ) Harmonic source: e(t) = Page 7 / 40 L k=1 e jξ k(t), where dξ k dt = 2πf k (t) Quasi-stationarity assumption: ξ k (t τ) ξ k (t) 2πf k (t)τ Filtered signal: x(t) = L k=1 M(t, f k (t)) e j(ξ k (t)+ϕ(t,f k (t))) McAulay and Quatieri model (speech coding) x(t) = L k=1 A k(t) e jψ k (t) where dψ k dt = 2πf k (t) and A k (t) and f k (t) have slow variations compared with e jψ k(t) Serra and Smith model (music signal synthesis) x(t) = L k=1 A k(t) e jψ k (t) + b(t) where b(t) is a white noise filtered by a time-varying filter Complete analysis / modification / synthesis system: estimation of the deterministic components linear interpolation of amplitudes and cubic interpolation of phases subtraction of the deterministic part to get b(t) transformation of each of the two components re-synthesis Page 8 / 40

3 Scales modifications Equivalence of the two modifications Duration modification Duration modification Temporal distortion function: τ = T (t) Modified signal: y(τ) = L k=1 A k(t 1 (τ)) e jφ k (τ) Preservation of the frequencies: φ k (τ) = 2π τ 0 f k(t 1 (u))du Pitch modification Spectral compression rate: α(t) Modified signal: y(t) = L k=1 A k(t) e jφ k (t) Frequencies modification: Φ k (t) = 2π t 0 α(u)f k (u)du Pitch modification Reciprocity temporal distortion T plus temporal re-scaling T 1 pitch modification of rate α(t) = T (t) Page 9 / 40 Page 10 / 40 Principle diagram Part II Short time Fourier transform Page 11 / 40 Page 12 / 40

4 Short time Fourier transform Equivalent band-pass filter Definition: X(t a, ν) = n Z x(n + t a ) w a (n) e j2πνn, where the analysis window w a (n) is finite, real and symmetric the analysis times t a are indexed by an integer u Interpretation: band-pass convention X(t a, ν p ) = [x h](t a ) where h(n) = w a ( n) e j2πν pn the FT h(n) is H(e j2πν ) = W a ( e j2π(ν p ν) ) Discrete version of the STFT: let ν p = p N X(t a, ν p ) = N 1 pn j2π x(n + t a ) w a (n) e N n=0 the length of the analysis window must be N Page 13 / 40 Page 14 / 40 Synthesis diagram Signal reconstruction Perfect reconstruction condition (t s = t a and Y = X) Overlap-add (OLA) synthesis y(n) = u w s(n t s (u)) y w (n t s (u), t s (u)) supp(w s ) [0, N 1], y w (n, t s (u)) = 1 N 1 N p=0 Y (t s(u), ν p ) e j2πν pn sufficient condition: u w a(n t a (u)) w s (n t a (u)) 1 Modifications and problems raised: Modification of the amplitudes and phases of the STFT t a t s, X(t a (u), ν p ) Y (t s (u), ν p ) Difficulty: Y is generally not the STFT of a signal Re-synthesis from a sinusoidal model Page 15 / 40 Page 16 / 40

5 Instantaneous frequency Part III Phase vocoder McAulay and Quatieri model: x(t) = L k=1 A k(t) e jψ k (t) { Quasi-stationarity assumption: n {0... N 1} Ak (n + t a ) A k (t a ) Ψ k (n + t a ) Ψ k (t a ) + 2πf k (t a )n Then X(t a (u), ν p ) = L k=1 A k(t a ) e jψ k(t a ) ( W a e j2π(ν p f k (t a )) ) Let f c be the cutting frequency of the low-pass filter w a (n) Narrow band condition:! l such that ν p f l (t a ) f c Interpretation (harmonic spectrum): N 4 f 0 Then X(t a (u), ν p ) = A l (t a ) e jψ l(t a ) W a ( e j2π(ν p f l (t a )) ) the STFT permits us to estimate phases Ψ l (t a ) modulo 2π Page 17 / 40 Page 18 / 40 Overlap condition Duration modification Removing the phase ambiguity modulo 2π: Phase difference between two successive times: Φ p = 2π(f l (t a ) ν p ) t a (u) + 2πν p t a (u) + 2nπ Minimal overlap condition: f c t a (u) < 1 2 Interpretation (Hann window): f c = 2 N t a < N 4! n such that Φ p 2πν p t a (u) 2nπ < π Estimation of the instantaneous frequency p {0... N 1} 1. computation of the STFT at two successive times Φ p 2. computation of Q(n 0 ) = Φ p 2πν p t a 2n 0 π such that Q(n 0 ) < π 3. computation of the instantaneous frequency f l (t a ) = ν p + Q(n 0) 2π t a Unwrapping of the instantaneous phases for a distortion T (t) Modification algorithm: 1. computation of the STFT and of f l (t a (u)) in each channel 2. computation of the new synthesis time t s (u) = T (t a (u)) 3. computation of the synthesis instantaneous phase Φ s (t s (u + 1), ν p ) = Φ s (t s (u), ν p ) + 2πf l (t a (u)) (t s (u + 1) t s (u)) 4. computation of the synthesis STFT at u + 1 Ỹ (t s (u + 1), ν p ) = A p (t a (u + 1)) e jφ s(t s (u+1),ν p ) Page 19 / 40 Page 20 / 40

6 Phase unwrapping Influence of the initial phases x(t) = H a k cos(2πk t T + ψ k) k=1 Page 21 / 40 Page 22 / 40 Pitch modification Temporal re-sampling method 1. time stretching of rate T (t) = t 0 α(u)du 2. temporal re-scaling of rate T 1 (τ) Spectral re-sampling method 1. Linear interpolation of the analysis STFT α(t a ) > 1: information loss in high frequencies α(t a ) < 1: spectral completion in high frequencies 2. re-synchronization of the phases in the re-synthesis Part IV Processing specific to speech Problem in speech processing: "Donald Duck" effect spectral envelope estimation (LPC) and "whitening" pitch modification, then inverse filtering Page 23 / 40 Page 24 / 40

7 Time-frequency reciprocity x( t α ) = H k=1 a k cos(2π k αt t + φ k) Time-frequency reciprocity x( t α ) = H k=1 a k cos(2π k αt t + φ k) Page 25 / 40 Page 26 / 40 Pitch modification of speech Case of unvoiced sounds Page 27 / 40 Page 28 / 40

8 Timbre and spectral envelope Pitch modification Voiced sounds: modify the fundamental frequency Voiced/unvoiced sounds: leave the spectral envelope unchanged Use of the vocoder 1. Signal whitening by filtering (LPC analysis) 2. Frequency scale modification 3. Inverse filtering Methods specific to monophonic speech signals Voiced/unvoiced segmentation Pitch estimation on the voiced frames Page 29 / 40 Page 30 / 40 Temporal modifications Part V TD-PSOLA Page 31 / 40 Page 32 / 40

9 Spectral modifications Example of pitch modification Comparison phase vocoder / PSOLA Page 33 / 40 Page 34 / 40 Speech production mechanism Part VI Voiced sounds: vibration of the vocal cords filtered by the vocal tract Unvoiced sounds: turbulent noise filtered by the vocal tract Auto-regressive models Page 35 / 40 Page 36 / 40

10 Production of voiced sounds Production of unvoiced sounds = = = Glottal pulses Vocal tract Voiced sound = Turbulent noise Vocal tract Unvoiced sound Page 37 / 40 Page 38 / 40 Signal model The vocal tract is modeled by an AR filter h(z) = a 1 z a p z p estimated by linear prediction (LPC analysis) Source model depending on the voiced / unvoiced case The glottal pulse train is modeled by an impulse train of period T s(t) = n δ(t nt ) Synthesis with auto-regressive models Synthesis without modification by overlap/add of the time frames convolution of the source with the filter on every frame Synthesis with modification Duration modification Synthesis of a source of appropriate length Pitch modification Unvoiced frames: unchanged Voiced frames: the period of the impulse train is changed The turbulent noise is modeled by a white noise Page 39 / 40 Page 40 / 40