Regularizing inverse problems using sparsity-based signal models

Transcription

1 Regularizing inverse problems using sparsity-based signal models Jeffrey A. Fessler William L. Root Professor of EECS EECS Dept., BME Dept., Dept. of Radiology University of Michigan fessler Work with Sai Ravishankar and Raj Nadakuditi CSP Seminar 31 Mar / 45

2 Disclosure Research support from GE Healthcare Supported in part by NIH grants P01 CA-87634, U01 EB Equipment support from Intel Corporation 2 / 45

3 Outline Why Low-dose X-ray CT imaging Accelerated MR imaging Other inverse problems How MAP estimation for inverse problems Classical regularization methods (some sparsity based) Contemporary regularization methods (all sparsity based) 3 / 45

4 Outline Why Low-dose X-ray CT imaging Accelerated MR imaging Other inverse problems How MAP estimation for inverse problems Classical regularization methods (some sparsity based) Contemporary regularization methods (all sparsity based) 4 / 45

5 Improving X-ray CT image reconstruction A picture is worth 1000 words (and perhaps several 1000 seconds of computation?) Thin-slice FBP ASIR (denoise) Statistical Seconds A bit longer Much longer Today s talk: less about computation, more about image quality Right image used edge-preserving regularization 5 / 45

6 Accelerating MR imaging (a) 4 under-sampled MR k- space (b) zero-filled reconstruction (c) compressed sensing reconstruction with TV regularization (d) adaptive dictionary learning regularization [1, Fig. 10] 6 / 45

7 Other ill-posed inverse problems y = Ax + ε compressed sensing (A random, wide) deblurring (restoration) (A Toeplitz, wide?) in-painting (A subset of rows of I) denoising (not ill posed) (A = I) 7 / 45

8 Outline Why Low-dose X-ray CT imaging Accelerated MR imaging Other inverse problems How MAP estimation for inverse problems Classical regularization methods (some sparsity based) Contemporary regularization methods (all sparsity based) 8 / 45

9 Outline Why Low-dose X-ray CT imaging Accelerated MR imaging Other inverse problems How MAP estimation for inverse problems Classical regularization methods (some sparsity based) Contemporary regularization methods (all sparsity based) 9 / 45

10 Inverse problems via MAP estimation Unknown image x System model p(y x) Data y Estimator ˆx If we have a prior p(x), then the MAP estimate is: ˆx = arg max x p(x y) = arg max log p(y x) + log p(x). x For gaussian measurement errors and linear model: log p(y x) 1 2 y Ax 2 W where y 2 W = y W y and W 1 = Cov{y x} is known (A from physics, W from statistics) 10 / 45

11 Priors for MAP estimation If all images x are plausible (have non-zero probability) then p(x) e R(x) = log p(x) R(x) (from fantasy / imagination / wishful thinking) MAP regularized weighted least-squares (WLS) estimation: ˆx = arg max x = arg min x log p(y x) + log p(x) 1 2 y Ax 2 W + R(x) A regularizer R(x), aka log prior, is essential for high-quality solutions to ill-conditioned / ill-posed inverse problems. Why ill-posed? High ambitions / 45

12 Example of ill-conditioned inverse problem Two-pixel, two-ray X-ray tomography model: y 1 x 1 x 2 y 2 A = [ ] x = [ x1 x 2 ] cond ( A A ) 400 A is (roughly) square - somewhat typical 1 x 2 log-likelihood log p(y x): y 1 x / 45

13 Subspace model: Alternative to regularization Assuming x lies in a sufficiently low-dimensional subspace could make an inverse problem well conditioned. x 2 x 1 [ ] 1 Assume x = Dz where D = and z R 1 1 (z has only one nonzero element so very sparse!?) Estimate coefficient(s): [ ẑ ] = arg min z y ADz 2 2, then ˆx = Dẑ, 2 where B AD = and cond ( B B ) = 1 which is perfect! / 45

14 Why not use subspace models? Candès and Romberg (2005) [2] used 22 (noiseless) projection views, each with 256 samples = 5632 measured values, vs = unknown pixels 1 Shepp-Logan Phantom dimensional subspace Subspace representation (using pixel basis) is undesirably coarse. 14 / 45

15 Outline Why Low-dose X-ray CT imaging Accelerated MR imaging Other inverse problems How MAP estimation for inverse problems Classical regularization methods (some sparsity based) Contemporary regularization methods (all sparsity based) 15 / 45

16 Classical regularizers Tikhonov regularization (IID gaussian prior) Roughness penalty (Basic MRF prior) Sparsity in ambient space Edge-preserving regularization Total-variation (TV) regularization Black-box denoiser like NLM 16 / 45

17 Tikhonov regularization x 2 R(x) = β x 2 2 x 1 Colors show equivalent (normalized) prior p(x) / p(0) = e R(x) Equivalent to IID gaussian prior on x Makes any ill-conditioned / ill-posed problem well conditioned Ignores correlations between pixels 17 / 45

18 Sparsity regularization in ambient space x 2 R(x) = β x 0 = β j I {x j 0} x 1 Approximate Bayesian interpretation Non-convex IID = also ignores correlations 18 / 45

19 Sparsity regularization: convex relaxation x 2 R(x) = β x 1 = β j x j x 1 Equivalent to IID Laplacian prior on x Also ignores correlations 19 / 45

20 Correlation Shepp-Logan Pixel Neighbors (dithered) 2 1 Shepp-Logan Phantom 1.05 xj x j Caution: Shepp-Logan phantom [3] was designed for testing non-bayesian methods, not for designing signal models. Q: What causes the spread?? 20 / 45

21 Edge-preserving regularization Neighboring pixels tend to have similar values except near edges: x 2 R(x) = β j ψ(x j x j 1 ) Potential function ψ: 1 x x 1 ψ(t) t/δ Equivalent to improper prior (agnostic to DC value) Accounts for spatial correlations, but only very locally Used clinically now for low-dose X-ray CT image reconstruction 21 / 45

22 Total-variation (TV) regularization Neighboring pixels tend to have similar values except near edges ( gradient sparsity ): R(x) = βtv(x) = β Cx 1 = β x j x j 1 j x 2 Potential function ψ: 1 x 1 ψ(t) t Equivalent to improper prior (agnostic to DC value) Accounts for correlations, but only very locally Well-suited to piece-wise constant Shepp-Logan phantom! Used in many academic publications / 45

23 Black-box denoiser as a regularizer Noisy image Denoiser Denoised image Example: Non-local means (NLM) Corresponding regularizer [4] [6]: R(x) = β 1 2 x NLM(x) 2 2 Encourages self-consistency with denoised version of image No evident Bayesian interpretation Variable splitting can facilitate minimization [7]. 23 / 45

24 Many more regularizers / priors Transforms: wavelets, curvelets,... Markov random field models Graphical models / 45

25 Outline Why Low-dose X-ray CT imaging Accelerated MR imaging Other inverse problems How MAP estimation for inverse problems Classical regularization methods (some sparsity based) Contemporary regularization methods (all sparsity based) 25 / 45

26 Contemporary regularizers Convolutional sparsity Union of subspaces Sparse coding with dictionary manifolds? [8] 26 / 45

27 Convolutional sparsity Idea: x[ n] K h k [ n] z k [ n] k=1 where each h k [ n] is a FIR filter with h k = 1 and each coefficient image z k [ n] is sparse [9] [11]. Equivalent matrix-vector representation: K x H k z k k=1 where H k is a Toeplitz (or circulant) matrix corresponding to h k. 27 / 45

28 Convolutional sparsity: example 0.5 Filter h 1 [n] 2 1 Coefficients z 1 [n] h 2 [n] 2 1 z 2 [n] 0.5 x[n] n 0.5 h 3 [n] 2 1 z 3 [n] x[ n] K k=1 h k [ n] z k [ n] / 45

29 Convolutional sparsity: regularizer Recall x K k=1 H k z k Natural corresponding regularizer: R(x) = min {z k } min {h k } h k = 1 K x 2 H k z k + λ 2 K z k 0 k=1 2 k=1 Adapts FIR filters {h k } and coefficients {z k } to candidate x. Literature focuses on the minimization problem (sparse coding) Yet to be explored as regularizer for inverse problems Inherently shift-invariant representation; no patches needed 29 / 45

30 Union of subspaces model x 2 x 1 Dimensionality reduction? cf. classification / clustering motivation [12] (Extension to union of flats (linear varieties) is possible [13].) 30 / 45

31 Union of subspaces regularization Given (?) collection of K subspace bases D 1,..., D K (dictionaries with full column rank): R(x) = min k }{{} classification = min k β 1 2 min z k β 1 2 x D kz k 2 2 } {{ } regression x D k D + k x 2 2 R(x) = 0 if x lies in the span of any of the dictionaries {D k }. otherwise, distance to nearest subspace (discourage, not constrain) Non-convex (highly?) (cf. preceding picture) Apply to image patches to be practical Equivalent Bayesian interpretation? (not a mixture model here) 31 / 45

32 Regularizers using sparse coding with dictionaries Assume x Dz where D is a dictionary (often over-complete) and z is a sparse coefficient vector. Corresponding regularizers: R(x) = min z : z p s β1 2 x Dz 2 2 ( ) 1 R(x) = min β 1 z 2 x Dz β 2 z p Convex in z (for given x) if p 1. R(x) typically non-convex in x. Could be equivalent to a union-of-subspaces regularizer if D = [D 1... D K ] and if we constrain coefficient vector z in a non-standard way. 32 / 45

33 Union-of-subspaces vs sparse-coding-with-dictionary Consider union-of-subspaces model with D 1 = So D 1 spans x-y plane and D 2 spans z-axis. A dictionary model with D = [D 1 D 2 ] = [ [ ], D 2 = and sparsity s = 2, happily represents all three cardinal planes ] [ ]. Thus dictionary model seems less constrained than union-of-subspaces model. (Still, focus on sparse dictionary representation hereafter.) 33 / 45

34 Dictionary learning from training data Given training data x 1,..., x N R d (image patches) Assumed model: x n Dz n unknown d J dictionary D = [d 1... d J ] coefficient vectors z 1,..., z N R J assumed sparse K-SVD dictionary learning formulation [14]: D = arg min D R d J = arg min D R d J N min x n Dz n 2 z n R J n=1 min X DZ F Z R J N s.t. s.t. d j = 1 j z n 0 s n d j = 1 j z n 0 s n X [x 1... x N ], Z [z 1... z N ] Computationally expensive and no convergence guarantees. Inherently non-convex due to product of unknowns DZ. 34 / 45

35 New dictionary learning method (SOUP-DIL) Joint work with Sai Ravishankar and Raj Nadakuditi [15] [18] Write sparse representation as Sum of OUter Products (SOUP): X DZ = DC = J j=1 d j c j where Z = C = [c 1... c J ] R N J (coefficients for each atom) Replace individual atom sparsity constraint z n 0 s with aggregate sparsity regularizer: Z 0 = C 0. Natural for DIctionary Learning (DIL) from training data Unnatural for image compression using sparse coding SOUP-DIL l 0 formulation: D = arg min D R d J min X DC 2 C R N J F + λ 2 C 0 s.t. d j 2 = 1 j c j L j 35 / 45

36 SOUP-DIL algorithm SOUP-DIL formulation: D = arg min D R d J min X DC 2 C R N J F + λ 2 C 0 s.t. d j 2 = 1 j c j L j Block coordinate descent (BCD) algorithm Sparse coding step for C Dictionary update step for D Very simple update rules (low compute cost) Monotone descent of Ψ(D, C) Convergence theorem: for any given initialization (D 0,C 0 ), all accumulation points of sequence (D,C) are critical points of cost Ψ and are equivalent { (reach same cost function value Ψ ). D Furthermore: (k) D (k 1) } { 0. Same for C (k)}. 36 / 45

37 SOUP-DIL updates D = arg min D R d J min X DC 2 C R N J F + λ 2 C 0 s.t. d j 2 = 1 j c j L j Alternate: update one column d j of D then one column c j of C. Sparse coding step: update c j using residual E j k j d kc k min c j E j d j c j 2 F + λ 2 c j 0 s.t. c j L Truncated (via L) hard thresholding of E jd j with threshold λ Dictionary atom step: update d j min d j E j d j c j 2 F s.t. d j 2 = 1 Constrained least-squares solution: d j = (E j c j )/ E j c j 2 37 / 45

38 Truncated hard thresholding for SOUP-DIL L c j R N λ λ L E jd j R N (Acts element-wise.) (In practice take L =.) (Algorithm also provides a simple sparse coding method.) 38 / 45

39 Example: dictionary learning for Barbara Barbara K-SVD D SOUP-DIL D Denoising PSNR (db) from [15] σ Noisy O-DCT K-SVD SOUP-DIL SOUP-DIL faster than K-SVD 39 / 45

40 Regularization using SOUP-DIL Large image x, extract M patches X = [P 1 x... P M x] Assume patch x m = P m x Dz m has (aggregate) sparse representation in dictionary D R d J where d is patch size R(x) = R(X) = min C R M J X DC 2 F +λ 2 C 0 s.t. c j L j R(x) = 0 if patches can be represented exactly with sufficiently few non-zero coefficients (depends on λ) Ignore constraint c j L Bayesian interpretation? 40 / 45

41 CT reconstruction using (known) dictionary regularizer ˆx = arg min x = arg min x 1 2 y Ax 2 W + β R(x) 1 min C R M J 2 y Ax 2 W + β ) ( X DC 2 F + λ 2 C 0 2 Alternating (nested) minimization: Fixing x, updating each column of C sequentially involves (truncated?) hard-thresholding Fixing C, updating x is (large-scale) quadratic problem g(x) = 1 2 X DC 2 F = M m=1 1 2 P m x DP m C 2 2 Work in progress... M 2 g(x) = P mp m m=1 is diagonal 41 / 45

42 Why Bayesian? Numerous normal-dose CT images! learn D, or most of it, from big data learn statistics of sparse coefficients Z? replace generic Z 0 with p(z)? 42 / 45

43 Extensions / future work Use majorization to update multiple columns of D or C simultaneously DC atom Rotate/flip atoms [19] [20] rank constraints on dictionary atoms [16] Tensor structured atoms for 3D / dynamic imaging Combined transform learning / dictionary learning Union of manifolds instead of union of subspaces?... Open problems Model selection Parameter selection Performance guarantees 43 / 45

44 Bibliography I [1] S. Ravishankar and Y. Bresler, MR image reconstruction from highly undersampled k-space data by dictionary learning, IEEE Trans. Med. Imag., vol. 30, no. 5, , May [2] E. Candès and J. K. Romberg, Signal recovery from random projections, in Proc. SPIE 5674 Computational Imaging III, 2005, [3] L. A. Shepp and B. F. Logan, The Fourier reconstruction of a head section, IEEE Trans. Nuc. Sci., vol. 21, no. 3, 21 43, Jun [4] M. Mignotte, A non-local regularization strategy for image deconvolution, Pattern Recognition Letters, vol. 29, no. 16, , Dec [5] Z. Yang and M. Jacob, Nonlocal regularization of inverse problems: A unified variational framework, IEEE Trans. Im. Proc., vol. 22, no. 8, , Aug [6] H. Zhang, J. Ma, J. Wang, Y. Liu, H. Lu, and Z. Liang, Statistical image reconstruction for low-dose CT using nonlocal means-based regularization, Computerized Medical Imaging and Graphics, vol. 38, no. 6, , Sep [7] S. Y. Chun, Y. K. Dewaraja, and J. A. Fessler, Alternating direction method of multiplier for tomography with non-local regularizers, IEEE Trans. Med. Imag., vol. 33, no. 10, , Oct [8] S. Poddar and M. Jacob, Dynamic MRI using smoothness regularization on manifolds (SToRM), IEEE Trans. Med. Imag., [9] J. Yang, K. Yu, and T. Huang, Supervised translation-invariant sparse coding, in Proc. IEEE Conf. on Comp. Vision and Pattern Recognition, 2010, [10] H. Bristow, A. Eriksson, and S. Lucey, Fast convolutional sparse coding, in Proc. IEEE Conf. on Comp. Vision and Pattern Recognition, 2013, [11] B. Wohlberg, Efficient convolutional sparse coding, in Proc. IEEE Conf. Acoust. Speech Sig. Proc., 2014, / 45

45 Bibliography II [12] R. Vidal, Subspace clustering, IEEE Sig. Proc. Mag., vol. 28, no. 2, 52 68, Mar [13] T. Zhang, A. Szlam, and G. Lerman, Median K-Flats for hybrid linear modeling with many outliers, in Proc. Intl. Conf. Comp. Vision, 2009, [14] M. Aharon, M. Elad, and A. Bruckstein, K-SVD: an algorithm for designing overcomplete dictionaries for sparse representation, IEEE Trans. Sig. Proc., vol. 54, no. 11, , Nov [15] S. Ravishankar, R. R. Nadakuditi, and J. A. Fessler, Efficient sum of outer products dictionary learning (SOUP-DIL) - The l 0 method, IEEE Trans. Sig. Proc., 2015, Submitted. [16], Efficient learning of dictionaries with low-rank atoms, in Proc. IEEE Intl. Conf. on Image Processing, Submitted., [17], Efficient l0 dictionary learning with convergence guarantees, in Proc. Intl. Conf. Mach. Learn, Submitted., [18],Efficient sum of outer products dictionary learning (SOUP-DIL) - The l 0 method, arxiv , [19] B. Wen, S. Ravishankar, and Y. Bresler, FRIST - flipping and rotation invariant sparsifying transform learning and applications of inverse problems, [20] B. Wen, Y. Bresler, and S. Ravishankar, FRIST - flipping and rotation invariant sparsifying transform learning and applications, arxiv , / 45