The rise and fall of regularizations: l1 vs. l2 representer theorems

Transcription

1 The rise and fall of regularizations: l1 vs. l2 representer theorems Michael Unser Biomedical Imaging Group EPFL, Lausanne, Switzerland Joint work with Julien Fageot, John-Paul Ward, and Harshit Gupta London Workshop on Sparse Signal Processing, Sept. 2016, Imperial College. OUTLINE Linear inverse problems and regularization Tikhonov regularization The sparsity (r)evolution Compressed sensing and l1 imization Part I: Discrete-domain regularization (l2 vs. l1) Part II: Continuous-domain regularization (L2 vs. gtv) Classical L2 regularization: theory of RKHS Splines and operators Minimization of gtv: the optimality of splines Enabling components for the proof Special case TV in 1D 2

2 Inverse problems in bio-imaging Linear forward model noise y = Hs + n H s Integral operator n Problem: recover s from noisy measurements y The easy scenario Basic limitations Inverse problem is well posed if 9c 0 > 0 s.t., for all s 2 X, c 0 ksk applekhsk ) s H 1 y Backprojection (poor man s solution): 1) Inherent noise amplification 2) Difficulty to invert H (too large or non-square) 3) All interesting inverse problems are ill-posed s H T y 3 Linear inverse problems (20th century theory) Dealing with ill-posed problems: Tikhonov regularization R(s) =klsk 2 2 : regularization (or smoothness) functional L: regularization operator (i.e., Gradient) s R(s) subject to ky Hsk 2 2 apple 2 Equivalent variational problem Andrey N. Tikhonov ( ) s? = arg ky Hsk klsk 2 2 {z } {z } data consistency regularization Formal linear solution: s =(H= T H + L T L) 1 H T y = R y Interpretation: filtered backprojection 4

3 Linear inverse problems: The sparsity (r)evolution (20th Century) p =2! 1 (21st Century) s rec = arg s ky Hsk R(s) Non-quadratic regularization regularization R(s) =klsk 2`2! klsk p`p! klsk`1 Total variation (Rudin-Osher, 1992) R(s) =klsk`1 with L: gradient Wavelet-domain regularization (Figuereido et al., Daubechies et al. 2004) v = W 1 s: wavelet epansion of s (typically, sparse) R(s) =kvk`1 Compressed sensing/sampling (Candes-Romberg-Tao; Donoho, 2006) 5 Compressive sensing (CS) and l1 imization y A [Donoho et al., 2005 Candès-Tao, 2006,...] + noise Sparse representation of signal: s = W with kk 0 = K N Equivalent N y N sensing matri : A = HW Constrained (synthesis) formulation of recovery problem kk 1 subject to ky Ak 2 2 apple 2 6

4 CS: Three fundamental ingredients (Donoho, IEEE T. Inf. Theo. 2006) (Candès-Romberg, Inv. Prob. 2007) 1. Eistence of sparsifying transform (W or L) - Wavelet basis - Dictionary - Differential operator (Gradient) 2. Incoherence of sensing matri A - Restricted isometry; few linearly dependent columns (spark) - Quasi-random and delocalized structure: Gaussian matri with i.i.d. entries, random sampling in Fourier domain 3. Non-linear signal recovery (l 1 imization) 7 CS: Eamples of applications in imaging - Magnetic resonance imaging (MRI) (Lustig, Mag. Res. Im. 2007) - Radio Interferometry - Teraherz Imaging (Wiau, Notic. R. Astro. 2007) (Chan, Appl. Phys. 2008) - Digital holography (Brady, Opt. Epress 2009; Marim 2010) - Spectral-domain OCT (Liu, Opt. Epress 2010) - Coded-aperture spectral imaging (Arce, IEEE Sig. Proc. 2014) - Localization microscopy (Zhu, Nat. Meth. 2012) - Ultrafast photography (Gao, Nature 2014) 8

5 Part I: Discrete-domain regularization 9 Classical regularized least-squares estimator Linear measurement model: y m = hh m, i + n[m], m =1,...,M System matri of size M N : H =[h 1 h M ] T LS = arg 2R N ky Hk kk 2 2 ) LS =(H T H + I N ) 1 H T y MX = H T a = a m h m where a =(HH T + I M ) 1 y m=1 Interpretation: LS 2 span{h m } M m=1 Lemma (H T H + I N ) 1 H T = H T (HH T + I M ) 1 10

6 Generalization: constrained l2 imization Discrete signal to reconstruct: =([n]) n2z Sensing operator H:`2(Z)! R M 7! z =H{} =(h, h 1 i,...,h, h M i) with h m 2 `2(Z) Closed conve set in measurement space: C R M Eample: C y = {z 2 R M : ky zk 2 2 apple 2 } Representer theorem for constrained `2 imization (P2) 2`2(Z) kk2`2 s.t. H{} 2 C The problem (P2) has a unique solution of the form MX LS = a m h m =H {a} m=1 with epansion coefficients a =(a 1,,a M ) 2 R M. (U.-Fageot-Gupta IEEE Trans. Info. Theory, Sept. 2016) 11 Constrained l1 imization sparsifying effect Discrete signal to reconstruct: =([n]) n2z Sensing operator H:`1(Z)! R M 7! z =H{} =(h, h 1 i,...,h, h M i) with h m 2 `1(Z) Closed conve set in measurement space: C R M Representer theorem for constrained `1 imization (P1) V = arg 2`1(Z) kk`1 s.t. H{} 2 C is conve, weak*-compact with etreme points of the form KX sparse [ ] = a k [ n k ] with K = k sparse k 0 apple M. k=1 V If CS condition is satisfied, then solution is unique (U.-Fageot-Gupta IEEE Trans. Info. Theory, Sept. 2016) 12

7 Controlling sparsity Measurement model: y m = hh m,i + n[m], m =1,...,M sparse = arg 2`1(Z) MX m=1 y m hh m,i 2 + kk`1! Sparsity Inde (K) Conv. DCT CS de (K) λ a): Sparse model 50 Conv. DCT 40 CS Geometry of l2 vs. l1 imization Prototypical inverse problem ky Hk 2`2 + kk 2`2, kk`2 subject to ky Hk 2`2 apple 2 ky Hk 2`2 + kk`1, kk`1 subject to ky Hk 2`2 apple 2 2 C y 1 = h T 1 y 2 1 `2-ball: = C 2 `1-ball: = C 1 14

8 Geometry of l2 vs. l1 imization Prototypical inverse problem ky Hk 2`2 + kk 2`2, kk`2 subject to ky Hk 2`2 apple 2 ky Hk 2`2 + kk`1, kk`1 subject to ky Hk 2`2 apple 2 2 C y 1 = h T 1 sparse etreme points 1 `2-ball: = C 2 `1-ball: = C 1 Configuration for non-unique `1 solution 15 Part II: Continuous-domain regularization 16

9 Part II: Continuous-domain regularization Photo courtesy of Carl De Boor 17 Continuous-domain regularization (L2 scenario) Z Regularization functional: klfk 2 L 2 = Lf() 2 d R d L: suitable differential operator Theory of reproducing kernel Hilbert spaces hf,gi H = hlf,lgi (Aronszajn 1950) Interpolation and approimation theory Smoothing splines (Schoenberg 1964, Kimeldorf-Wahba 1971) Thin-plate splines, radial basis functions (Duchon 1977) Machine learning Radial basis functions, kernel methods Representer theorem(s) (Poggio-Girosi 1990) (Schölkopf-Smola 2001) 18

10 Representer theorem for L2 regularization (P2) arg f2h! MX y m f( m ) 2 + kfk 2 H m=1 h : R d R d! R is the (unique) reproducing kernel for the Hilbert space H(R d ) if (i) h( 0, ) 2 H for all 0 2 R d (ii) f( 0 )=hh( 0, ),fi H for all f 2 H and 0 2 R d Conve loss function: F : R M R M! R Sample values: f = f( 1 ),...,f( M ) (P2 ) arg f2h F (y, f)+ kfk2 H (Schölkopf-Smola 2001) Representer theorem for L 2 -regularization The generic parametric form of the solution of (P2 ) is MX f() = a m h(, m ) m=1 Supports the theory of SVM, kernel methods, etc. 19 Sparsity and continuous-domain modeling Compressed sensing (CS) Generalized sampling and infinite-dimensional CS Xampling: CS of analog signals (Adcock-Hansen, 2011) (Eldar, 2011) Splines and approimation theory L 1 splines Locally-adaptive regression splines Generalized TV (Fisher-Jerome, 1975) (Mammen-van de Geer, 1997) (Steidl et al. 2005; Bredies et al. 2010) Statistical modeling Sparse stochastic processes (Unser et al ) 20

11 Geometry of l2 vs. l1 imization Prototypical inverse problem ky Hk 2`2 + kk 2`2, kk`2 subject to ky Hk 2`2 apple 2 ky Hk 2`2 + kk`1, kk`1 subject to ky Hk 2`2 apple 2 2 C y 1 = h T 1 y Geometry of L2 vs. TV imization Prototypical inverse problem s ky H{s}k 2`2 + kl{s}k 2 L 2, s kl{s}k 2 L 2 subject to ky H{s}k 2`2 apple 2 s ky H{s}k 2`2 + kl{s}k TV, s kl{s}k TV subject to ky H{s}k 2`2 apple 2 2 C y y 1 = hh 1,si L L-splines with M fied knots (H: pure sampling operator) 2 1 L-splines with few adaptive knots (H: can be arbitrary) 22

12 Splines are analog and intrinsically sparse L{ }: admissible differential operator ( 0 ): Dirac impulse shifted by 0 2 R d Definition The function s : R d! R is a (non-uniform) L-spline with knots ( k ) K k=1 if KX L{s} = a k ( k )=w : spline s innovation k=1 Spline theory: (Schultz-Varga, 1967) a k L= d d k k+1 FRI signal processing: Innovation variables (2K) (Vetterli et al., 2002) Location of singularities (knots) : { k } K k=1 Strength of singularities (linear weights): {a k } K k=1 23 Spline synthesis: eample L=D= d d D () =D 1 { }() = Null space: N D = span{p 1 }, p 1 () =1 + (): Heaviside function w () = X k a k ( k ) 1 s() =b 1 p 1 ()+ X k a k + ( k ) b 1 a 1 24

13 Spline synthesis: generalization L: spline admissible operator (LSI) L () =L 1 { }: Green s function of L Finite-dimensional null space: N L = span{p n } N 0 n=1 Spline s innovation: w () = X k a k ( k ) ) s() = X k a k L ( k )+ XN 0 n=1 b n p n () Requires specification of boundary conditions a 1 25 Principled operator-based approach Biorthogonal basis of N L =span{p n } N 0 n=1 =( 1,, N 0 ) such that h m,p n i = m,n XN 0 Projection operator: p = h n,pip n n=1 Operator-based spline synthesis for all p 2 N L Boundary conditions: hs, ni = b n,n=1,,n 0 Spline s innovation: L{s} = w = X k a k ( k ) s() =L 1 {w }()+ XN 0 n=1 b n p n () Eistence of L 1 as a stable right-inverse of L? (see Theorem 1) LL 1 w = w (L 1 w)=0 26

14 Beyond splines: function spaces for gtv Photo courtesy of Carl De Boor 27 From Dirac impulses to Borel measures S(R d ): Schwartz s space of smooth and rapidly decaying test functions on R d S 0 (R d ): Schwartz s space of tempered distributions Space of real-valued, countably additive Borel measures on R d M(R d )= C 0 (R d ) 0 = w 2 S 0 (R d ):kwk M = sup hw, 'i < 1, '2S(R d ):k'k 1 =1 where w : ' 7! hw, 'i = R R d '(r)w(r)dr Equivalent definition of total variation norm kwk M = sup hw, 'i '2C 0 (R d ):k'k 1 =1 Basic inclusions ( 0 ) 2 M(R d ) with k ( 0 )k M =1for any 0 2 R d kfk M = kfk L1 (R d ) for all f 2 L 1 (R d ) ) L 1 (R d ) M(R d ) 28

15 Optimality result for Dirac measures F: linear continuous map M(R d )! R M C: conve compact subset of R M Generic constrained TV imization problem V = arg w2m(r d ):F(w)2C kwk M Generalized Fisher-Jerome theorem The solution set V is a conve, weak -compact subset of M(R d ) with etremal points of the form KX w = a k ( k ) k=1 with K apple M and k 2 R d. (U.-Fageot-Ward, ArXiv 2016) Jerome-Fisher, 1975: Compact domain & scalar intervals 29 General conve problems with gtv regularization M L (R d )= s : gtv(s) =kl{s}k M = sup hl{s}, 'i < 1 k'k 1 apple1 Linear measurement operator M L (R d )! R M : f 7! z =H{f} C: conve compact subset of R M Finite-dimensional null space N L = {q 2 M L (R d ):L{q} =0} with basis {p n } N 0 n=1 Admissibility of regularization: H{q 1 } =H{q 2 }, q 1 = q 2 for all q 1,q 2 2 N L Representer theorem for gtv regularization The etremal points of the constrained imization problem V = arg f2m L (R d ) kl{f}k M s.t. H{f} 2 C KX XN 0 are necessarily of the form f() = a k L ( k )+ b n p n () with K apple k=1 M N 0 ; that is, non-uniform L-splines with knots at the k and kl{f}k M = P k=1 a k. The full solution set is the conve hull of those etremal points. n=1 V (U.-Fageot-Ward, ArXiv 2016) 30

16 Enabling components for proof of the theorem 31 Eistence of stable right-inverse operator L 1,n0 (R d )={f : R d! R : sup f() (1 + kk) n 0 < +1} 2R d Theorem 1 (U.-Fageot-Ward, ArXiv 2016) Let L be a spline-admissible operator with a N 0 -dimensional null space N L L 1,n0 (R d ) such that p = P N 0 n=1 hp, nip n for all p 2 N L. Then, there eists a unique and stable operator L 1 : M(R d )! L 1,n0 (R d ) such that, for all w 2 M(R d ), Right-inverse property: LL 1 w = w, Boundary conditions: (L 1 w)=0 with =( 1,, N 0 ). Its generalized impulse response g (, y) =L 1 { ( y)}() is given by g (, y) = L ( y) XN 0 n=1 p n ()q n (y) with L such that L{ L } = and q n (y) =h n, L ( y)i. 32

17 Characterization of generalized Beppo-Levi spaces Regularization operator L:M L (R d )! M(R d ) f 2 M L (R d ), gtv(f) =kl{f}k M < 1 Theorem 2 (U.-Fageot-Ward, ArXiv 2016) Let L be a spline-admissible operator that admits a stable right-inverse L 1 of the form specified by Theorem 1. Then, any f 2 M L (R d ) has a unique representation as f =L 1 w + p, where w =L{f} 2 M(R d ) and p = P N 0 n=1 h n,fip n 2 N L with n 2 M L (R d ) 0. Moreover, M L (R d ) is a Banach space equipped with the norm kfk L, = klfk M + k (f)k 2. Generalized Beppo-Levi space: M L (R d )=M L, (R d ) N L M L, (R d )= f 2 M L (R d ): (f) =0 N L = p 2 M L (R d ):L{p} =0 33 Eample: Conve problem with TV regularization L=D= d d N D = span{p 1 }, p 1 () =1 D () = + (): Heaviside function General linear-inverse problem with TV regularization kd{s}k M s.t. H{s} =(hh 1,si,, hh M,si) 2 C(y) s2m D (R) Generic form of the solution (by Theorem 4) KX s() =b 1 + a k + ( k ) no penalty k=1 b 1 a 1 with K<Mand free parameters b 1 and (a k, k ) K k=1 34

18 SUMMARY: Sparsity in infinite dimensions Discrete-domain formulation Contrasting behavior of l 1 vs. l 2 regularization Minimization of l 1 favors sparse solutions (independently of sensing matri) Continuous-domain formulation Linear measurement model Linear signal model: PDE Ls = w s 2 X s 7! z =H{s} ) s =L 1 w L-splines = signals with sparsest innovation gtv(s) =klsk M Deteristic optimality result gtv regularization: favors sparse innovations Non-uniform L-splines: universal solutions of linear inverse problems 35 Acknowledgments Many thanks to (former) members of EPFL s Biomedical Imaging Group Dr. Pouya Tafti Prof. Arash Ai Dr. Emrah Bostan Dr. Masih Nilchian Dr. Ulugbek Kamilov Dr. Cédric Vonesch... and collaborators... Prof. Demetri Psaltis Prof. Marco Stampanoni Prof. Carlos-Oscar Sorzano Dr. Arne Seitz... Preprints and demos:

19 References New results on sparsity-promoting regularization M. Unser, J. Fageot, H. Gupta, Representer Theorems for Sparsity-Promoting `1 Regularization, IEEE Trans. Information Theory, Vol. 62, No. 9, pp M. Unser, J. Fageot, J.P. Ward, Splines Are Universal Solutions of Linear Inverse Problems with Generalized-TV Regularization, arxiv: [math.fa]. Theory of sparse stochastic processes M. Unser and P. Tafti, An Introduction to Sparse Stochastic Processes, Cambridge University Press, Preprint, available at For splines: see chapter 6 Algorithms and imaging applications E. Bostan, U.S. Kamilov, M. Nilchian, M. Unser, Sparse Stochastic Processes and Discretization of Linear Inverse Problems, IEEE Trans. Image Processing, vol. 22, no. 7, pp , C. Vonesch, M. Unser, A Fast Multilevel Algorithm for Wavelet-Regularized Image Restoration, IEEE Trans. Image Processing, vol. 18, no. 3, pp , March M. Nilchian, C. Vonesch, S. Lefkimmiatis, P. Modregger, M. Stampanoni, M. Unser, Constrained Regularized Reconstruction of X-Ray-DPCI Tomograms with Weighted-Norm, Optics Epress, vol. 21, no. 26, pp , Link with Total variation of Rudin-Osher Total variation of function 6= total variation of a measure Total variation in 1D: TV(f) =sup k'k1 apple1hdf,'i = kdfk M perfect equivalence (with L=D) Usual total variation in 2D: TV(f) =sup k'k1 apple1hrf,'i Problem: r y ) is not a scalar operator L 1 version of the 2D total variation: Z TV(f) = rf(, y) ddy, = 1 R 4 2 Z 2 0 kd u fk M d angular averaging (rotation invariance) D u f = hu, rfi: directional derivative of f along u = (cos, sin ) Present theory eplains the regularisation effect of kd u fk M 38