Sparse stochastic processes: A statistical framework for modern signal processing

Transcription

1 Sparse stochastic processes: A statistical framework for modern signal processing Michael Unser Biomedical Imaging Group EPFL, Lausanne, Switzerland Plenary talk, Int. Conf. Syst. Sig. Im. Proc. (IWSSIP), Bucharest, July 7-9, 03 0th century statistical signal processing Hypothesis: Signal = stationary Gaussian process Karhunen-Loève transform (KLT) is optimal for compression DCT asymptotically equivalent to KLT (Ahmed-Rao, 97; U., 98) (Pearl et al., IEEE Trans. Com 97)

2 0th century statistical signal processing (cont d) Hypothesis: Signal = Gaussian process y = Hs + n Noise: i.i.d. Gaussian with variance Signal covariance: C s = E{s s T } Wiener filter is optimal for restoration/denoising s LMMSE = C s H T HC s H T + I y = FWiener y Wiener (LMMSE) solution = Gauss MMSE = Gauss MAP s MAP = arg min s ky Hsk {z } Data Log likelihood + kc / s sk {z } Gaussian prior likelihood, quadratic regularization (Tikhonov) 3 Then came wavelets... and sparsity Alfred Haar 90 // 98 Stéphane Mallat Ingrid Daubechies Sparsity 006 Applications Compressed sensing Martin Vetterli David Donoho Emmanuel Candès

3 Fact : Wavelets can outperform Wiener filter w = T (w) w Fact : Wavelet coding can outperform jpeg f(x) = X i,k i,k(x) w i,k Wavelet transform Inverse wavelet transform Discarding small coefficients (Shapiro, IEEE-IP 993) 6

4 Fact 3: l schemes can outperform l optimization s? = argmin ky Hsk + R(s) {z } {z } data consistency regularization Wavelet-domain regularization Wavelet expansion: s = Wv (typically, sparse) Wavelet-domain sparsity-constraint: R(s) with v = W s =kvk` Iterated shrinkage-thresholding algorithm (ISTA, FISTA) (Nowak et al., Daubechies et al. 00) ` regularization (Total variation) R(s) =klsk` with L: gradient (Rudin-Osher, 99) Iterative reweighted least squares (IRLS) or FISTA 7 Quest for a unifying framework: the precursor Gaussian stationary processes as a filtered white noise s(t) =(h w)(t) s(!) = H(!) w(!) / H(!) Z Frequency response (shaping filter): H(!) = h(t)e j!t dt Whitening operator Innovation: Ls(t) =w(t) L(!) = H(!) R w is Gaussian stationary and independent at every point 8

5 Continuous-domain innovation model Shaping filter Generalized white noise w(x) L { } (appropriate boundary conditions) Whitening operator L{ } Stochastic process s(x), x R d Our work: the non-gaussian part of this story Main finding: it is necessarily sparse (infinitely divisible) Why?... as explained by our current research... (invoking powerful theorems in functional analysis: Bochner-Minlos, Gelfand, Schoenberg & Lévy-Khinchine) 9 Innovation-based synthesis of splines (x) L= d dx =D L : integrator L { } =L Impulse response (x x 0 ) Translation invariance L { } (x x 0 ) X a[k] (x x k ) kz Linearity L { } s(x) = X kz a[k] (x x k ) 0

6 OUTLINE Sparse stochastic processes Generalized innovation model Gelfand s theory of generalized stochastic processes Statistical characterization of sparse stochastic processes Lévy processes and their generalization Fractal processes: Gaussian vs. sparse Applications Modeling of signals (audio) Algorithms for sparse signal recovery as MAP estimators Optimal denoising (MMSE) Sparse representations, optimal transforms Sparse stochastic processes

7 Road map for theory of sparse processes Specification of inverse operator Functional analysis solution of SDE 3 Characterization of generalized stochastic process White noise w L Mixing operator Whitening operator L s =L w Very easy! (after solving. &.) Multi-scale wavelet analysis Characterization of continuous-domain white noise Higher mathematics: generalized functions (Schwartz) measures on topological vector spaces Gelfand s theory of generalized stochastic processes Infinite divisibility (Lévy-Khintchine formula) Characterization of transform-domain statistics Easy when: i =L i 3 Generalized innovation process Difficulty : w 6= w(x) is too rough to have a pointwise interpretation Difficulty : w is an infinite-dimensional random entity; its pdf can be formally specified by a measure P w (E) where E S 0 (R d ) Axiomatic definition (Gelfand-Vilenkin 96) w is a generalized innovation process (or continuous-domain white noise) over S 0 (R d ) if. Observability : X = hw, 'i is a well-defined random variable for any test function ' S(R d ).. Stationarity : X x0 = hw, '( x 0 )i is identically distributed for all x 0 R d. 3. Independent atoms : X = hw, ' i and X = hw, ' i are independent whenever ' and ' have non-intersecting support. X = h, i X = h, i Characteristic functional (!! ' ) Z dp w (') =E{e jhw,'i } = e jhs,'i P w (ds) S 0

8 Defining Gaussian noise: discrete vs. continuous Lévy exponent:! log p X (!) = f (!) = Discrete white Gaussian noise X = (X,..., XN ) with Xn i.i.d standardized Gaussian Characteristic function: p X (!) = g(!) = exp PN n= k!k f (!n ) = e Continuous-domain white Gaussian noise 0 Infinite-dimensional entity w with generic observations Xn = hw, 'n i d Characteristic functional: P w (') = G(') = e k'kl = exp cw (!'n ) = e p Xn (!) = E{ej!hw,'n i } = E{ejhw,!'n i } = P Z R f '(x) dx! k'n kl Characterization of generalized innovation X' = hw, 'i = h lim h = i, lim h, n! n! i + + h, i,, i Theorem Let w be a generalized stochastic process such that Xid = hw, recti is well-defined. Then, w is a generalized innovation (white noise) over S 0 (Rd ) if and only if its char- acteristic form is given by jhw,'i d P } = exp w (') = E{e Z Rd f '(x) dx where f (!) is a valid Le vy exponent (in fact, the Le vy exponent of Xid ). Moreover, the random variables X' = hw, 'i are all infinitely divisible with modified Le vy exponent f' (!) = Z Rd f!'(x) dx (Gelfand-Vilenkin 96; Amini-U. under review) 6

9 Examples of id noise distributions Observations: pid (x) Xn = hw, rect( (a) Gaussian n)i ! f ( ) = log +! f (!) = (b) Laplace Sparser (c) Compound Poisson 0.7 R f ( ) = 0. 0 R (ejx )p(x)dx (d) Cauchy (stable) 0.30 f (!) = s! d Complete mathematical characterization: P w ( ) = exp Z Rd (x) dx f 7 Steps + 3: Characterization of sparse process L s=l White noise w w Whitening operator L Abstract formulation of innovation model s=l ) S, w, s =, L cs (') = E{ejhs,'i } = P d P w (L ') = exp w = L, w {z } Z Rd f L '(x) dx Sufficient condition for existence: L continuous operator: S(Rd )! Lp (Rd ) (U.-Tafti-Sun, preprint ArXiv 0) 8

10 Example : Lévy processes Ds = w (unstable SDE!) s =D0 w S,,s = D0,w D 0 '(t) = Z t 0 '( )d White noise (innovation) Lévy process Gaussian Brownian motion 0 0 (Wiener 93) Integrator Impulsive w(t) Z t 0 d s(t) Compound Poisson S S (Cauchy) Lévy flight (Paul Lévy circa 930) 9 Example : Self-similar processes L F! (j!) H+ ) L : fractional integrator fbm; H = 0.0 Poisson; H = 0.0 fbm; H = 0.7 fbm; H =. fbm; H =.0 H=. H=.7 H=. H=. Poisson; H = 0.7 Poisson; H =. Poisson; H =.0 Gaussian Sparse (generalized Poisson) Fractional Brownian motion (Mandelbrot, 968) (U.-Tafti, IEEE-SP 00) 0

11 Scale- and rotation-invariant processes Stochastic partial differential equation : ( ) H+ s(x) =w(x) Gaussian H=. H=.7 H=. H=.7 Sparse (generalized Poisson) (U.-Tafti, IEEE-SP 00) A brief panorama of applications

12 A. Signal modeling (Audio) Sparse, bandpass processes L= dn dt n + a n dt n + + a d dt + a 0I d n (a) Gaussian (b) Alpha stable α=. Mixed sparse processes: s mix = s + + s M cp smix (') = MY m= cp sm (') =exp Z MX R m= f m Lm '(t) dt! Gaussian (Am) generalized Lévy (Am, SαS) 3 A. Biomedical imaging: MAP reconstruction Innovation model of the signal Ls = w s = L w Signal decoupling: discrete version of operator u(x) =L d s(x), u = Ls (matrix notation) Generalized B-spline L =L d L Statistical characterization - X =[u] n identically distributed (approx. independent) - Probability density function: p X (x) =F { d P w (! L )}(x) - Potential function: X(x) = log p X (x) Maximum a posteriori (MAP) estimator for AWN s = argmin kg Hsk + P n X([Ls] n ) (Bostan et al., IEEE-IP 03)

13 Sparser MAP estimator: special cases s = argmin kg Hsk + P n X([Ls] n ) Gaussian: p X (x) = 0 e x /( 0 ) X(x) = x 0 Laplace: p X (x) = e x X(x) = x Student: p X (x) = B r, r+ x + X(x) = r + log( + x ) Student potentials: r =,, 8, 3 (fixed variance) MRI: Shepp-Logan phantom Original SL Phantom Fourier Sampling Pattern Angles L : gradient Optimized parameters Laplace prior (TV) Student prior (log)

14 MRI reconstruction Real T Brain Image MR Angiography Image k-space sampling pattern 0 radial lines Reconstruction results in db L : gradient Optimized parameters Gaussian Estimator Laplace Estimator Student s Estimator T brain Image MR Angiography Image D deconvolution experiment Astrocytes cells bovine pulmonary artery cells human embryonic stem cells Disk shaped PSF (7x7) Deconvolution results in db L : gradient Optimized parameters Gaussian Estimator Laplace Estimator Student s Estimator Astrocytes cells Pulmonary cells Stem cells (Bostan et al., IEEE-IP 03)

15 A3. Optimal denoising: MMSE formulation Measurement model: y = x + n n: additive white Gaussian noise x: sparse first-order process (e.g., Lévy flight) u = Ls: discrete innovation (i.i.d. and infinite divisible) p Y X (y x ) p Y X (y x ) p Y X (y 3 x 3 ) Optimal estimator: x MMSE = E{x y} 6 µ X (x) µ + X (x) p(x y) = Z NY Y N p Y X y n x n n= n= p U x n x n {z } u n X 3 X X 3 p U (x ) p U (x x ) p U (x 3 x ) Belief propagation algorithm SNR [db] 3 MMSE Log = MAP TV LMMSE Lévy process with Cauchy increments AWGN n (Kamilov et al., IEEE-SP 03) 9 A. Sparse representations, optimal transforms Innovation model (SDE) Ls = w s = L w Admissible basis function: i,k =L i,k with i,k L p (R d ) Signal expansion = equivalent white-noise analysis Y = h i,k,si = hl i,k, L wi = h i,k,wi =) ˆp Y (!) = d P w (! i,k ) Statistical implications Transform-domain pdfs are infinitely divisible Quality of decoupling depends upon support of wavelet/smoothing kernel i,k 30

16 Operator-like wavelets for sparse AR() processes Innovation model: Ls = w, s =L w with L = (D I) Operator-like wavelet: i =L i with i: smoothing kernel (Khalidov-U., 006) Wavelet analysis: hs, i ( t 0 )i = hl w, L i( t 0 )i = hw, i( t 0 )i 0 =L 0 Haar! 0 (x) =L (a) (b) 3 Orthogonal expansion of a SαS Lévy process Mutual Information 0 α sparser Gaussian (Pad-U. ICASSP 3, SPARS 3)

17 Orthogonal expansion of a SαS AR() process Mutual Information Identity DCT/KLT Haar Wavelet Operator like Wavelet Optimal (ICA) α (Pad-U. ICASSP 3, SPARS 3) sparser Gaussian e =0.9, M = 6 CONCLUSION! Unifying continuous-domain innovation model! Backward compatibility with classical Gaussian theory! Operator-based formulation: Lévy-driven SDEs! Gaussian vs. sparse (generalized Poisson, student, SαS)! Focus on unstable SDEs non-stationary, self-similar processes! Wavelet analysis vs. regularization! Central role of B-spline (see papers)! Sparsification/decoupling via operator-like behavior! Theoretical framework for sparse signal recovery! Analytical determination of PDF in any transformed domain! Predictive power: transform coding/denoising (facts,, 3)! New statistically-founded sparsity priors! Derivation of estimators (MAP vs. MMSE): link with LASSO and l methods for sparse signal recovery (Compressed sensing) 3

18 References Theory of generalized stochastic processes M. Unser, P. Tafti, and Q. Sun, A unified formulation of Gaussian vs. sparse stochastic processes Part I: Continuous-domain theory, preprint, available at M. Unser, P. Tafti, A. Amini, H. Kirshner, A Unified Formulation of Gaussian vs. Sparse Stochastic Processes Part II: Discrete-Domain Theory, preprint, available at M. Unser, P.D. Tafti, Stochastic models for sparse and piecewise-smooth signals, IEEE Trans. Signal Processing, vol. 9, no. 3, pp , March 0. Q. Sun, M. Unser, Left-Inverses of Fractional Laplacian and Sparse Stochastic Processes, Advances in Computational Mathematics, vol. 36, no. 3, pp. 399-, April 0. Recent applications U.S. Kamilov, P. Pad, A. Amini, M. Unser, MMSE Estimation of Sparse Lévy Processes, IEEE Trans. Signal Processing, vol. 6, no., pp. 37-7, 03. A. Amini, U.S. Kamilov, E. Bostan, M. Unser, Bayesian Estimation for Continuous-Time Sparse Stochastic Processes, IEEE Transactions on Signal Processing, vol. 6, no., pp , 03. E. Bostan, U.S. Kamilov, M. Nilchian, M. Unser, Sparse Stochastic Processes and Discretization of Linear Inverse Problems, IEEE Trans. Image Processing, in press. P. Pad, M. Unser, On the Optimality of Operator-Like Wavelets for Sparse AR() Processes, Proc. IEEE Int. Conf. Acoust. Speech Sig. Proc. (ICASSP 3), Vancouver, Canada, May, 03, in press. 3 Acknowledgments Many thanks to " Dr. Pouya Tafti " Prof. Qiyu Sun " Prof. Thierry Blu " Emrah Bostan " Dr. Matthieu Guerquin-Kern " Dr. Arash Amini " Dr. Hagai Kirshner " Pedram Pad " Ulugbek Kamilov " Masih Nilchian " Members of EPFL s Biomedical Imaging Group! Preprints and demos: 36

19 ebook: web preprint 37