Some aspects of Time-Frequency multipliers

Transcription

1 Some aspects of Time-Frequency multipliers B. Torrésani LATP, Université de Provence, Marseille, France MulAc kickoff meeting, September 28

2 The group in Marseille LMA : Olivier Derrien, Richard Kronland-Martinet, Adrien Merer, Thibaud Necciari, Anaik Olivero, LATP : Adil El Moudni, Anaik Olivero, Bruno Torrésani Works in collaboration with L. Daudet (LAM, Paris 7), Ph. Depalle (McGill Montreal), C. Benar (INSERM Marseille),...

3 Projects and collaborations Currently, 3 projects running in Marseille centered around multipliers : MulAc project : Multipliers and masking Operator approximation using multipliers and generalizations McGill/LMA/LATP project : (with Ph Depalle, McGill, Montreal) Multiplier estimation Masking pursuit Sound morphing PEPS project : (with L. Daudet, Paris 7) Multiplier estimation Sound genealogy

4 Integral kernel/matrix Spreading function Affine spreading function 1 Introduction 2 Integral kernel/matrix Spreading function Affine spreading function 3 using Gabor multipliers using wavelet multipliers Generalized multipliers 4 Estimating a multiplier Applications Estimating MGM 5

5 Kernel and matrix Integral kernel/matrix Spreading function Affine spreading function In L 2 (R) a linear operator H is often characterized by its integral kernel κ H, via Hf (t) = κ H (t, s)f (s) ds with integral and equality defined according to the context. In finite dimensional situations (i.e. C N ), the role of the kernel is played by the matrix (w.r.t. the canonical basis) Hf [m] = N 1 n= K[m, n]f [n] In both cases, alternate representations are provided by matrices with respect to suitable bases.

6 Spreading function Integral kernel/matrix Spreading function Affine spreading function The idea : expand the operator with respect to operator building blocks, here time-frequency shifts M ξ T t, with M ξ f (t) = e 2iπξt f (t), T τ f (x) = f (t τ) Proposition Let H H, the class of Hilbert-Schmidt operator on L 2 (R). Then there exists a function η = η H L 2 (R 2 ), called the spreading function such that H = η(t, ξ)m ξ T τ dτdξ.

7 Spreading function (2) Integral kernel/matrix Spreading function Affine spreading function The spreading function is intimately related to the integral kernel κ = κ H of H via η H (τ, ξ) = and κ H (t, s) = the correspondence being isometric κ H (t, t τ)e 2iπξt dt η H (t s, ξ)e 2iπξt dξ, η H 2 = κ H 2 = H H

8 Integral kernel/matrix Spreading function Affine spreading function Spreading function in concrete situations In C N (with periodic boundary conditions), translations and modulations are defined as M ξ f [t] = e 2iπξt/N f [t], T τ f [t] = f [t τ], t, τ, ξ =,... N 1 all operations being understood modulo N. Proposition Let H End(C N ). Then there exists a spreading function η = η H C N N, such that H = N 1 τ,ξ= η[τ, ξ]m ξ T τ.

9 Integral kernel/matrix Spreading function Affine spreading function Spreading function in concrete situations (cont.) Again, the isometry property still holds η[τ, ξ] = N 1 t= κ H [t, t τ]e 2iπξτ/N. and κ H [t, s] = N 1 ξ= η H [t s, ξ]e 2iπξt/N, with η H = κ H = H F

10 Affine spreading function Integral kernel/matrix Spreading function Affine spreading function Question : does there exist a similar tool when modulation is replaced with dilation : expand operators in terms of shifts T τ and rescalings : D a f (t) = 1 ( t f, a a) Answer : not exactly, almost...

11 Affine spreading function Integral kernel/matrix Spreading function Affine spreading function Question : does there exist a similar tool when modulation is replaced with dilation : expand operators in terms of shifts T τ and rescalings : D a f (t) = 1 ( t f, a a) Answer : not exactly, almost... Let D = ( ) 1/4 = F 1 ν F, where is the Laplacian. Associate with any η L 2 (R R +, da dτ/a 2 ) the operator da dτ H η = η(τ, a)t τ D a D R R + a 2,

12 Affine spreading function (cont.) Integral kernel/matrix Spreading function Affine spreading function An explicit calculation gives the integral kernel of H η where the kernel reads H η f (t) = κ(t, s) = D acting on the first variable of η. κ(t, s)f (s) ds Dη(t as, a) da a,

13 Affine spreading function (cont.) Integral kernel/matrix Spreading function Affine spreading function Proposition The transform η H η is an isometry H η 2 H = η 2 L 2 (R R +,da dτ/a 2 ), and is therefore invertible by its adjoint, η(b, a) = H η, π(b, a)d = Trace(H η D π(b, a) ). In other words, one may obtain the affine spreading function from the (Fourier transform of the) integral kernel of the operator η(τ, a) = a ˆκ(ν, aν) νe 2iπντ dν

14 Discrete affine spreading function Integral kernel/matrix Spreading function Affine spreading function Questions : 1 Does there exist simple discrete versions of affine spreading function? 2 Which version of discrete wavelet transform should one choose? Wavelet frame Littlewood-Paley type representation Wavelet on discrete field... Since groups seem to be an important aspect of the spreading function business, the wavelet transform on Z p, with p a prime number, is probably the best candidate.

15 using Gabor multipliers using wavelet multipliers Generalized multipliers 1 Introduction 2 Integral kernel/matrix Spreading function Affine spreading function 3 using Gabor multipliers using wavelet multipliers Generalized multipliers 4 Estimating a multiplier Applications Estimating MGM 5

16 Multiplier approximation using Gabor multipliers using wavelet multipliers Generalized multipliers Heuristics : when the spreading function (whichever version) is concentrated enough, the operator doesn t involve too large displacements, and multiplier approximations should perform well. Multiplier : Given two Gabor systems {g mn, m, n Z}, {h mn, m, n Z}, with lattice constants (b, ν ), and an upper symbol (or time-frequency transfer function, or mask) m, the corresponding Gabor multiplier reads M m f = m(m, n) f, g mn h mn m,n= (with a similar expression in the finite case) Notation : V g f (b, ν) = f, π(b, ν)g = f (t)g(t b)e 2iπνt dt

17 Multiplier approximation (cont.) using Gabor multipliers using wavelet multipliers Generalized multipliers A simple example : set τ = 1/ν and ξ = 1/b under suitable assumptions on the windows and lattice constants, the optimal Gabor multiplier approximation of H H is given by M (τ, ξ) = k,l= V g h (τ + kτ, ξ + lξ ) η H (τ + kτ, ξ + lξ ) k,l= V g h (τ + kτ, ξ + lξ ) 2 where M is the symplectic Fourier transform of the transfer function M (τ, ξ) = m= n= m(m, n)e 2iπ(nν τ mb ξ).

18 Multiplier approximation (cont.) using Gabor multipliers using wavelet multipliers Generalized multipliers where m(m, n) = = M (τ, ξ)e 2iπ(nν τ mb ξ) dτdξ ] τ 2, τ [ ] ξ 2 2, ξ [ 2 is the fundamental domain of the adjoint lattice.

19 Multiplier approximation (cont.) using Gabor multipliers using wavelet multipliers Generalized multipliers Quality of the approximation : let M m the best Gabor multiplier approximation of H H. Then where we set V = V g h, H M m 2 H η H 2 1 E k,l η H(τ + kτ, ξ + lξ )V (τ + kτ, ξ + lξ ) 2 E(τ, ξ) = U (τ, ξ) k,l η H(τ + kτ, ξ + lξ ) 2, and U (τ, ξ) = k,l V (τ + kτ, ξ + lξ ) 2

20 Multiplier approximation (cont.) using Gabor multipliers using wavelet multipliers Generalized multipliers Underspread operators : assume that Supp(η h ) ] τ 2, τ [ 2 ] ξ 2, ξ [ 2 then η H (τ, ξ)v (τ, ξ) 2 E(τ, ξ) = U (τ, ξ) η H (τ, ξ) 2 = V (τ, ξ) 2 k,l V (τ + kτ, ξ + lξ ) 2 To minimize the error 1 E : minimize the overlap between shifted copies of V.

21 Multiplier approximation (cont.) using Gabor multipliers using wavelet multipliers Generalized multipliers The same, in the discrete case : set N b = N/N b, N ν = N/N ν The optimal Gabor multiplier approximation of H End(C N ) is given by M (τ, ξ) = Nν 1 Nb 1 k= l= V g h (τ + kτ, ξ + lξ ) η H (τ + kτ, ξ + lξ ) Nb 1 l= V g h (τ + kτ, ξ + lξ ) 2 Nν 1 k= where M is the (finite) symplectic Fourier transform of the transfer function M (τ, ξ) = N ν 1 m= N b 1 n= m(m, n)e 2iπ(nν τ mb ξ)/n. Then : very fast algorithms, compatible with real life signals.

22 Multiplier approximation (cont.) using Gabor multipliers using wavelet multipliers Generalized multipliers Next question : for a given operator, a given analysis Gabor system, and a given transfer function, can one find the optimal synthesis window? currently, this seems possible in theory, but our preliminary results seem quite unstable...

23 Multiplier approximation (cont.) using Gabor multipliers using wavelet multipliers Generalized multipliers Next question : for a given operator, a given analysis Gabor system, and a given transfer function, can one find the optimal synthesis window? currently, this seems possible in theory, but our preliminary results seem quite unstable... An important remark : the phase of the transfer function m plays a crucial role in the behavior of the multiplier. From the previous discussion, it follows that the optimal Gabor multiplier approximations always produce complex-valued transfer functions. Complex oscillations of the transfer function produce (small) time-frequency shifts.

24 Wavelet multiplier approximation using Gabor multipliers using wavelet multipliers Generalized multipliers One may think that the affine spreading function representation is the appropriate language for studying wavelet multiplier approximation. Wavelet multipliers are not connected with the affine spreading function as simply as Gabor multipliers.

25 Wavelet multiplier approximation using Gabor multipliers using wavelet multipliers Generalized multipliers One may think that the affine spreading function representation is the appropriate language for studying wavelet multiplier approximation. Wavelet multipliers are not connected with the affine spreading function as simply as Gabor multipliers. For example, the (classical) spreading function of a STFT multiplier takes the remarkably simple form η Mm (τ, ξ) = M (τ, ξ)v g h(τ, ξ), i.e. factorizes into a product where the first term only depends on the mask, and the second term only depends on the windows. Here, M (τ, ξ) = is defined on the whole delay-doppler plane. m(b, ν)e 2iπ(ντ bξ) dbdν

26 Wavelet multiplier approximation using Gabor multipliers using wavelet multipliers Generalized multipliers In the wavelet case : no such simple decoupling, the simplest expression reads η(τ, a) = ( ) τ (1 a)τ a m(τ, a da dτ )W ψ Dψ a, a a 5/2 where W ψ denotes continuous wavelet transform with wavelet ψ, and D is as before. In the discrete case : same questions as before...

27 Generalized multipliers using Gabor multipliers using wavelet multipliers Generalized multipliers Multipliers essentialy represent operators whose spreading function is concentrated enough. To represent operators whose spreading function is more spread out : Gabor multipliers can be generalized in several ways : MGM : Multiple Gabor Multipliers (linear combinations of Gabor multipliers on a fixed grid) : see Monika s lecture... Sparsity : does it make sense to seek sparse linear combinations of time-frequency projection operators? for example, minimize something like H 2 α k π(z k )h g + µ α 1 k where the time-frequency points z k are limited to a (fine) grid????

28 Estimating a multiplier Applications Estimating MGM 1 Introduction 2 Integral kernel/matrix Spreading function Affine spreading function 3 using Gabor multipliers using wavelet multipliers Generalized multipliers 4 Estimating a multiplier Applications Estimating MGM 5

29 Multiplier estimation Estimating a multiplier Applications Estimating MGM The model problem : given several realizations of a transformation x (k) x (k) 1, k = 1,... K supposed to be linear, when can one approximate such a transformation by a time-frequency multiplier (possibly in some generalized sense), and estimate the corresponding transfer function from data? [ ] min m,a k k 1 A k M m x (k) 2 + Φ[m] x (k) where Φ is some regularization term, introducing prior information on the multiplier.

30 Multiplier estimation (cont) Estimating a multiplier Applications Estimating MGM For given priors Φ (i.e. l 2 or l 1 norm of m or 1 m), the problem may be solved (to some point... see Anaik s talk) Question : which information should one put in the prior?

31 Multiplier estimation (cont) Estimating a multiplier Applications Estimating MGM For given priors Φ (i.e. l 2 or l 1 norm of m or 1 m), the problem may be solved (to some point... see Anaik s talk) Question : which information should one put in the prior? Concretely : assume that the x (k) are realizations of a given trumpet sound, and the x (k) 1 are realizations of a similar clarinet sound, what do we want to assume on m?

32 Multiplier estimation (cont) Estimating a multiplier Applications Estimating MGM More complex estimation problem : condider the model problem : given several realizations of a transformation x (k) x (k) 1, k = 1,... K assume that each x (k) 1 is obtained from the x (k) via a time-frequency displacement combined with a Gabor multiplier. The problem is now to solve [ ] min x (k) 1 A k U k M m x (k) 2 + Φ[m] m,a k,u k k where U k is, say, a time-frequency shift. Again, Φ is some regularization term.

33 Multiple multiplier estimation Estimating a multiplier Applications Estimating MGM In a similar way, can one estimate a MGM : x (k) x (k) 1 j A j M mj x (k), k = 1,... K based upon a variational approach... more on that below

34 Multiple multiplier estimation Estimating a multiplier Applications Estimating MGM In a similar way, can one estimate a MGM : x (k) x (k) 1 j A j M mj x (k), k = 1,... K based upon a variational approach... more on that below Remark : as before, notice that the estimated transfer functions m are complex-valued.

35 Application : sound morphing Estimating a multiplier Applications Estimating MGM Problem : given two signals x, x 1, find what s in between... Using the model x 1 M m x, introduce x 1/2 = M 1/2 m x

36 Application : sound morphing Estimating a multiplier Applications Estimating MGM Problem : given two signals x, x 1, find what s in between... Using the model x 1 M m x, introduce x 1/2 = M 1/2 m x but what s the meaning of M 1/2 m, in particular when m is complex valued? even the naive choice M 1/2 m = M m raises the question of the choice of the determination of the square root. See Adrien s talk...

37 Application : sound genealogy Estimating a multiplier Applications Estimating MGM A slightly different problem : given a family of signals, can one find a way to estimate a hierarchical classification so that all signals are obtained from a common ancestor (a richer sound) by several Gabor multiplier operations

38 Application : sound genealogy Estimating a multiplier Applications Estimating MGM A slightly different problem : given a family of signals, can one find a way to estimate a hierarchical classification so that all signals are obtained from a common ancestor (a richer sound) by several Gabor multiplier operations

39 Application : sound genealogy (2) Estimating a multiplier Applications Estimating MGM For that : necessary to be able to estimate a common ancestor x for two signals, so that min m 1,m 2,{A (k) 1 },{A(k) 2 } x (k) 1 A (k) 1 M m 1 x (k), x (k) 2 A (k) 2 M m 2 x (k) again in a variational formulation [ k x (k) 1 A (k) 1 M m 1 x (k) + ] k x (k) 2 A (k) 2 M m 2 x (k) 2 + Φ[m 1, m 2 ] the question is what to put in the prior? 2

40 Estimating a multiplier Applications Estimating MGM Estimating Metro Goldwin Multipliers For transformations that cannot be accurately represented by Gabor multipliers : MGM

41 Estimating a multiplier Applications Estimating MGM Estimating Metro Goldwin Multipliers For transformations that cannot be accurately represented by Gabor multipliers : MGM Masking pursuit : a greedy algorithm to minimize quantities such as x (k) 1 2 A (k) j U j M mj x (k), k = 1,... K j k

42 Masking pursuit Estimating a multiplier Applications Estimating MGM Initialization : Set r () = x 1. Iteration : for n =,..N max 1, Estimate a mask m n+1 and time-frequency shifts (k n+1, l n+1 ) from x and residual r (n), using the approach developed above. Update the residual r (n+1) = r (n) π kn+1l n+1 M mn+1;g,g x This yields an estimate H N max 1 n= π knl n M mn;g,g for a MGM approximation of the operator

43 Masking pursuit (2) Estimating a multiplier Applications Estimating MGM Gabor Transform Magnitude of Signal u 1 Gabor Transform Magnitude of Signal u 2 Gabor Transform Magnitude of Signal u Frequency (Hz) Frequency (Hz) Frequency (Hz) Time (s) Time (s) Time (s) Gabor Transform Magnitude of Signal r 1 Gabor Transform Magnitude of Signal r 2 Gabor Transform Magnitude of Signal r Frequency (Hz) Frequency (Hz) Frequency (Hz) Time (s) Time (s) Time (s) Iteration 1 Iteration 2 Iteration 3

44 1 Introduction 2 Integral kernel/matrix Spreading function Affine spreading function 3 using Gabor multipliers using wavelet multipliers Generalized multipliers 4 Estimating a multiplier Applications Estimating MGM 5

45 Coming soon...