Sparse stochastic processes

Transcription

1 BIOMEDICAL IMAGING GROUP (BIG) LABORATOIRE D IMAGERIE BIOMEDICALE EPFL LIB Bât. BM CH 115 Lausanne Switzerland Téléphone : Fax : Site web : Michael.Unser@epfl.ch Dr. Michael Liebling Sparse stochastic processes Biological Imaging Center California Inst. of Technology Mail Code Pasadena, CA 91125, USA Part I: Introduction Lausanne, August 19, 24 Michael Unser Biomedical Imaging Group EPFL, Lausanne, Switzerland Dear Dr. Liebling, I am pleased to inform you that you were selected to the receive the 24 Research Award of the Swiss ofkorea Biomedical Engineering for your thesis work On Fresnelets, interference InvitedSociety seminar, Advanced Inst. of Science & Technology (KAIST), July 17-1, Seoul, Korea fringes, and digital holography. The award will be presented during the general assembly of the SSBE, September 3, Zurich, Switzerland. Please, lets us know if 1) you will be present to receive the award, 2) you would be willing to give a 1 minutes presentation of the work during the general assembly. 2th century statistical signal processing Hypothesis: Signal = stationary Gaussian process The award comes with a cash prize of 1.- CHF. Karhunen-Loève transform (KLT) is optimal for compression Would you please send your banking information to the treasurer of the SSBE, Uli Diermann ( uli.diermann@bfh.ch), so that he can transfer the cash prize to your account? I congratulate you on your achievement. DCT asymptotically equivalent to KLT With best regards, (Ahmed-Rao, 1975; U., 194) modulation the cross-modulationloss. improvement system can it conventional Michael Unser, Professor Chairman of the SSBE Award Committee cc: Ralph Mueller, president of the SSBE; Uli Diermann, treasurer Fig. 5. Rate versus distortion for various transforms for a second-order Markov process ( p =.95, N = 256). Gauss- (Pearl et al., IEEE Trans. Com 1972) 2

2 2th century statistical signal processing Hypothesis: Signal = Gaussian process y = Hs + n Noise: i.i.d. Gaussian with variance 2 Signal covariance: C s = E{s s T } Wiener filter is optimal for restoration/denoising s LMMSE = C s H T HC s H T + 2 I 1 y = FWiener y m L = Cs 1/2 : Whitening filter Wiener (LMMSE) solution = Gauss MMSE = Gauss MAP s MAP = arg min s 12 ky Hsk2 2 {z } Data Log likelihood + kc 1/2 s sk 2 2 {z } Gaussian prior likelihood, quadratic regularization (Tikhonov) klsk Then came wavelets... and sparsity Yves Meyer Alfred Haar 191 // 192 Stéphane Mallat 197- Ingrid Daubechies 1994 Sparsity 26 Applications Compressed sensing Martin Vetterli David Donoho Emmanuel Candès 4

3 Fact 1: Wavelets can outperform Wiener filter w = T (w) 2 w 5 Fact 2: Wavelet coding can outperform jpeg f(x) = X i,k i,k(x) w i,k Wavelet transform Inverse wavelet transform Discarding small coefficients (Shapiro, IEEE-IP 1993) 6

4 Fact 3: l1 schemes can outperform l2 optimization s? = argmin ky Hsk R(s) {z } {z } data consistency regularization Wavelet-domain regularization Wavelet expansion: s = Wv (typically, sparse) Wavelet-domain sparsity-constraint: R(s) =kvk`1 with v = W 1 s (Nowak et al., Daubechies et al. 24) `1 regularization (Total variation) (Rudin-Osher, 1992) R(s) =klsk`1 with L: gradient Compressed sensing/sampling (Candes-Romberg-Tao; Donoho, 26) 7 SPARSE STOCHASTIC MODELS: The step beyond Gaussianity Requirements for a comprehensive statistical framework Backward compatibility Continuous-domain formulation piecewise-smooth signals, translation and scale-invariance, sampling... Predictive power Can wavelets really outperform sinusoidal transforms (KLT)? Statistical justification and refinement of current algorithms Sparsity-promoting regularization, `1 norm minimization

5 1.3 FROM SPLINES TO STOCHASTIC PROCESSES Splines and Legos revisited Higher-order polynomial splines Random splines, innovations, Lévy processes Wavelet analysis of Lévy processes Lévy s synthesis of Brownian motion 9 Splines and Legos revisited Cardinal spline of degree : piecewise-contant f 1 (t) = X k2z f 1 [k] +(t k) +(t) = ( 1, for apple t<1, otherwise. Notion of D-spline: Df 1 (t) = X k2z a 1 [k] (t k) 1

6 B-spline and derivative operator Derivative Df(t) = df(t) dt D F! j! Finite difference operator D d f(t) =f(t) f(t 1) D d F! 1 e j! =( + Df)(t) B-spline of degree +(t) =D d D 1 (t) =D d + (t) l 1 +(t) = + (t) +(t 1) ˆ + (!) = 1 e j! j! Random splines and innovations cardinal Ds(t) = X n a n (t t n )=w(t) non-uniform Random weights {a n } i.i.d. and random knots {t n } (Poisson with rate ) Anti-derivative operators Shift-invariant solution: D 1 '(t) =( + ')(t) = Scale-invariant solution: D 1 '(t) = Z t '( )d Z t 1 '( )d 12

7 Innovation-based synthesis L= d dt =D ) L 1 : integrator (t) L 1 { } =L 1 (t t ) Translation invariance L 1 { } Impulse response (t t ) X a n (t t n ) n Linearity L 1 { } s(t) = X n a n (t t n ) 13 Compound Poisson process Stochastic differential equation Ds(t) =w(t) with boundary condition s() = Innovation: w(t) = X n a n (t t n ) Formal solution s(t) =D 1 w(t) =X n = X n a n D 1 { ( t n)}(t) a n + (t t n ) +( t n ) (impose boundary condition) 14

8 Generalization: Lévy processes Generalized innovations : white Lévy noise with E{w(t)w(t )} = 2 w (t t ) Ds = w (unstable SDE!) s =D 1 1 w, ' 2 S, h',si = hd ',wi White noise (innovation) Lévy process Gaussian Brownian motion (Wiener 1923) Integrator Impulsive w(t) Z t d s(t) Compound Poisson S S (Cauchy) Lévy flight (Paul Lévy circa 193) 15 Decoupling Lévy processes: increments Increment process: u(t) =D d s(t) =D d D 1 w(t) =( + w)(t). Increment process is stationary with autocorrelation function R u ( ) =E{u(t)u(t + )} = + ( +) _ R w (y) = 2 w 1 +( 1) * Discrete increments u[k] = s(k) s(k 1) = hw, +( k)i. u[k] are i.i.d. because { +( k)} are non-overlapping h, i w is independent at every point (white noise) 16

9 Wavelet analysis of Lévy processes Haar wavelets >< Haar(t) = >: 1, for apple t< 1 2 1, for 1 2 apple t<1, otherwise. >< i = >:,,2 i,k(t) =2 i/2 Haar t 2 i k 2 i >< i =1 >: 1, i =2 ( 2, 17 Wavelets as multi-scale derivatives >< i = >:,,2,,2 >< i = >: >< i =1 >: 1, >< i =1 >: 1, i =2 ( 2, i =2 ( 2, i,k =2 i/2 1 D i,k D 1 i,k =2 i/2 1 i,k. Wavelet coefficients of Lévy process Y i,k = hs, i,k i/hs, D i,k i /hd s, i,ki = hw, i,ki 1

10 M-term approximation: wavelets vs. KLT 1 Gaussian Identity KLT Haar Brownian motion Finite rate of innovation 1 1 Identity KLT Haar Compound Poisson Even sparser Identity KLT Haar Lévy flight (Cauchy) Wavelet-based synthesis of Brownian motion White Gaussian noise w = X X Z i,k i,k with Z i,k = hw, i,k i i2z k2z Brownian motion s(t) =D 1 w = X X Z i,k D 1 i,k(t) i2z k2z = X X 2 i/2 1 Z i,k i,k (t) i2z k2z 2

11 Unser and Tafti One puff to follow, one puff to follow, one puff to follow, one puff to follow, one puff to follow, one puff to follow, one puff to follow AUTHOR NAME, affiliation An Introduction to Sparse Stochastic Processes An Introduction to Sparse Stochastic Processes Unser and Tafti PPC. C M Y K Providing a novel approach to sparse stocastic processes, this comprehensive book presents the theory of stochastic process that are ruled by stochastic differential equations, and that admit a parsimonious representation in a matched wavelet-like basis. Two key themes are the statistical property of infinite divisibility, which leads to two distinct types of behavior Gaussian and sparse and the structural link between linear stochastic processes and spline functions, which is exploited to simplify the mathematical analysis. The core of the book is devoted to investigating sparse processes, including a complete description of their transform-domain statistics. The final part develops practical signal-processing algorithms that are based on these models, with special emphasis on biomedical image reconstruction. This is an ideal reference for graduate students and researchers with an interest in signal/image processing, compressed sensing, approximation theory, machine learning, or statistics. M I C H A E L U N S E R is Professor and Director of EPFL s Biomedical Imaging Group, Switzerland. He is a member of the Swiss Academy of Engineering Sciences, a fellow of EURASIP, and a fellow of the IEEE. Michael Unser and Pouya D. Tafti P O U YA D. TA F T I is a researcher at Qlaym BmbH, Düsseldort, and a former member of the Biomedical Imaging Group at EPFL, where he conducted research on the theory and applications of probablilistic models for data. Cover illustration: to follow ebook: web preprint EDEE Course 21 Extension to fractional orders: B-splines L = D (fractional derivative) Ld = + Fourier multiplier: (j!) (fractional differences) + (r) = + r+ Fourier multiplier: (1 1 F!... + (r) = +1 + r+ ( + 1) One-sided power function: e j! j! e j! j! e j! )... F! r+ = 1 ( +1 r, r, r< Degree = 1 22

12 Mandelbrot s fractional Brownian motion Solution of fractional SDE (Blu-U., 27) D s = w where w is Gaussian white noise ) s =I 2 w I 2 : scale-invariant, right-inverse of D Hurst exponent: H = Continuous-domain innovation model Shaping filter Generalized white noise w(x) L 1 { } (appropriate boundary conditions) Whitening operator L{ } Stochastic process s(x), x 2 R d Main outcome: non-gaussian solutions are necessarily sparse (infinitely divisible) Why?... as will explained in next chapters... (invoking powerful theorems in functional analysis: Bochner-Minlos, Gelfand, Schoenberg & Lévy-Khinchine) 24

13 2.1 DECOUPLING OF SPARSE PROCESSES s =L 1 w, w =Ls Discrete approximation of operator Operator-like wavelet analysis 25 Decoupling: Linear combination of samples Input: s(k), k 2 Z d (sampled values) s =L 1 w Discrete approximation of whitening operator: L d L d (x) = X k2z d d L [k] (x k) Discrete increment process: u[k] =L d s(x) x=k =( L w)(x) x=k = h L _ ( k),wi {z } ' Generalized B-spline: L(x) =L d L 1 (x) A-to-D translator 26

14 Decoupling: Wavelet analysis Ls = w Generalized operator-like wavelets: i(x) =L i(x) (Khalidov-U. 26, Ward-U. JFAA 213) Operator-like wavelet analysis of sparse process: h i ( x ),si =hl i( x ),si =h i ( x ), Ls)i =h i ( x ),wi =( _ i w)(x ) {z } ' 27