1 Wavelet Based Image Restoration Using Cross-Band Operators Erez Cohen Electrical Engineering Department Technion - Israel Institute of Technology Supervised by Prof. Israel Cohen
2 Layout Introduction - Wavelet-Based Image Restoration LTI Systems Representation in the WPT Domain Multiplicative Operators in the WPT Domain Image Restoration with Cross-band Filters Conclusion
3 Image Restoration True Image Observed Image Estimated Image 2 ( ) e~ N 0, σ e xmn (, ) amn (, ) ymn (, ) xmn ˆ(, )
4 Space Domain Image Restoration Quadratic Regularization ( ) h x Penalty (prior) function Convex optimization, closed form solution Linear space-invariant restoration Singular regions like edges, contain most of the perceptual information In low SNR, strong regularization smears these local features Nonquadratic Regularization Total variation, Gibbs distribution, etc. Compound GMRF ( ) ( ) m m= 1 i, i ( ( ) ) m ϕ t - edge preserving penalty function Nonlinear solution, sometimes even nonconvex target function. M Q x h x = ω ϕ 1 2 i, i 1 2
5 Wavelet Domain Image Restoration e x A y T W y% ˆx% W ˆx Related Work Expectation-Maximization (EM) Algorithm M. Figueiredo and R. Nowak, An EM algorithm for wavelet-based image restoration, IEEE Trans. Image Processing, vol. 12, no. 8, pp. 906 916, Aug. 2003. Gaussian Scale Mixtures (GSM) J. Bioucas-Dias, Bayesian wavelet-based image deconvolution: a GEM algorithm exploiting a class of heavy-tailed priors, IEEE Trans. Image Processing, vol. 15, no. 4, pp. 937 951, Apr. 2006. ForWaRD R. Neelamani, H. Choi, and R. Baraniuk, Forward: Fourier-wavelet regularized deconvolution for illconditioned systems, IEEE Trans. Signal Processing, vol. 52, no. 2, pp. 418 433, Feb. 2004.
6 Statistical Model of the Wavelet Coefficients Individual Wavelet Coefficient Heavy-tailed distributions describe the sparsity of wavelet representation. Few large coefficients dominate the signal s energy. Most of the coefficients are nearly zero. Heavy-tailed Gaussian Dependence Between Wavelet Coefficients Wavelet coefficients are roughly decorrelated. Although spatial and cross-scale dependence exists, statistical independence is often assumed. p l ( x% ) p( xp, p, k) = k= 1 p, p 1 2 1 2 Dependence between scales can be described with hidden Markov trees [Crouse et al. 1998].
7 Bayesian Restoration MAP estimation Wavelet-Based Image Restoration 2D-DFT domain Wavelet domain Unlike alone, is not block-circulant, and cannot be diagonalized by the 2D-DFT matrix The convolution matrix doesn t have a sparse structure For images of size, the convolution matrix size is
8 EM Algorithm for Wavelet-Based IR MPLE/MAP - Maximum Penalized Likelihood Estimator { p( ) pθ ( )} x% = arg max log y x% + x% x% 2 y AWx% = arg max + p 2 x% 2σ e θ ( x% ) A AW Unlike alone, is not block-circulant. An equivalent decoupled scheme, defining : e= αae + e ( 0, I) σ ( ) 2 1 2 2 2 T 2 e e2 ~ N ( 0, σe I α AA ) α T λmax AA e 1 N State model : z=wx% + αe y=ax+e y=az+ e2 1
9 EM Algorithm for Wavelet-Based IR Direct maximization : Expectation-Maximization : 2 y AWx% x% = arg max + p 2 θ % x% 2σ e ( x) () ( ˆ t () ) log ( ), ˆ t x, % x% z y, z x% y x% Q E p { Q( ˆ ) p θ ( )} ˆ( t+ 1) ( t) x% = arg max x,x % % + x% θ M-step Wavelet-base denoising E-step Fourier-base deconvolution ( ) 2 E-step: ˆ ˆ α x Wx% z = xˆ + A y Axˆ σ () t () t () t () t T () t 2 e 2 { α p θ ( )} () t () ˆ ( ) () M-step: T t t+ t % % arg max % % 2 % 1 2 z = W z x = x z x x%
10 Gaussian Scale Mixtures for Wavelet-Based IR Bayesian Restoration 2 y AWx% x% = arg max + p % 2 x% 2σ e ( x) Gaussian Scale Mixtures (GSM) prior ( ) ( ) ( ) p x = p x z pz z dz 0 xz ( ) xz~ N 0, z GEM Algorithm E-step: M-step: 1 () ( ) () () ( t+ 1) 2 ( t) () ( ) D d x = E z x t t t t p p, k p p, k 1, 2 1, 2 ( σ e D ) 1 T T T T x% = + WAAW W A y In order to implement the M-step, second-order stationary iterative method is applied
11 Fourier-Wavelet Regularized Deconvolution (ForWaRD) Deconvolution by transform-domain shrinkage y W 1 A T 1 x = x+ A e xˆ λ n 1 k = 0 1 ( ) xˆ = xb, + A eb, λ b λ k k k k Lower bound for MSE : 2 2 ( x bk σ k ) Small MSE only when most of the signal energy and colored noise energy are captured by just a few transform-domain coefficients. Fourier basis economically represents the colored stationary noise. n 1 MSE xˆ = min, + λ k = 0 Wavelet basis economically represents classes of signals that contain singularities...deconvolution techniques employing shrinkage in a single transform domain cannot yield adequate estimates in many deconvolution problems of interest.
12 Fourier-Wavelet Regularized Deconvolution (ForWaRD) ForWaRD y 1 A 1 x = x+ A e xˆ λ ˆx Employing scalar shrinkage both in the Fourier domain and in the wavelet domai First, ForWaRD employs a small amount of Fourier shrinkage to significantly attenuate the amplified noise components with a minimal loss of signal components. subsequent wavelet shrinkage in Step 2 effectively estimates the retained signal from the low-variance leaked noise. ForWaRD s hybrid approach yields robust solutions to a wide variety of deconvolution problems Signals with more economical wavelet representations should require less Fourier shrinkage.
13 Bayesian Restoration MAP estimation Wavelet-Based Image Restoration Unlike alone, is not block-circulant, and cannot be diagonalized by the 2D-DFT matrix The convolution matrix doesn t have a sparse structure For images of size, the convolution matrix size is 2D-DFT domain Wavelet domain
14 The Goal Wavelet-Based Image Restoration To find a sparse representation of LTI systems in the wavelet packet domain : Previous work Uniform decomposition [Banham et al. 1993] DTWT (Pyramid) decomposition [Zervakis et al. 1995] Our work Applicable for any admissible wavelet-packet decomposition Relies on cross-band filtering notation Can be easily extended to two-dimensions using separable bases
15 Layout Introduction - Wavelet-Based Image Restoration LTI Systems Representation in the WPT Domain Multiplicative Operators in the WPT Domain Image Restoration with Cross-band Filters Conclusion
16 Wavelet Packet Transform (WPT) Equivalent Nonuniform Filter-Bank
17 Representation of LTI Systems in the WPT Domain Time Domain WPT Domain x( n) h 1 ( n) 2 j1 x p,1 d p,1 2 j1 f 1 ( n) dn ( ) = xn ( ) an ( ) h 2 ( n) 2 j2 x p,2 ϕ d p,2 2 j2 f 2 ( n) k h ( n) j 2 k x p, k d p, k j 2 k f k ( n) l h ( n) 2 j l xp, l d p, l 2 j l f l ( n) ( K l ) d = ϕ x,, x p',1 k l pk, p',1 p',
18 Representation of LTI Systems in the WPT Domain Cross-band filtering notation x p,1 a p, k, p ',1 d pk,,1 Cross-band filter x p,2 a pk,, p ',2 d pk,,2 d pk, x pk, ' a pk,, p ', k ' d pkk,, ' xp, l a p, k, p', l d pk,, l Cross-band filters are used on system identification in the STFT domain for acoustic echo cancellation [Avargel and Cohen 2007] a = a pk,, p', k' nkk,, ' j k ( ) n= 2 p p' In the wavelet domain, the cross-band filters are time varying
19 Representation of LTI Systems in the WPT Domain Cross-band filtering notation d = x a pkk,, ' p', k' nkk,, ' p' j k j k' n= 2 p 2 p' Divide into 3 cases : j > j k k' x p, k ' a j k ' 2 p, kk, ' ' j 2 k j k d p, kk, ' j = j k k' x p, k ' a j k 2 pkk,, ' d p, kk, ' j < j k k' x p, k ' ' j 2 k j k a 2 j k pkk,, ' d p, kk, ' Any cross-term between two distinct subbands is a multirate filter, which depends on the subbands scales ratio Band-to-band terms between any subband to itself are always time-invariant filters
20 Representation of LTI Systems in the WPT Domain Cross-band filtering notation x p,1 a pk,,1 d pk,,1 x p,2 a p, k,2 d p, k,2 Band-to-band filter x p, k ' a pkk,, ' d p, kk, ' d pk, x p, k a pkk,, d p, kk, xp, l apk,, l d p, k, l Decimated cross-band filter pkk,, ' k k' ( ) ( ) ( ) k k' a = a n h n f n p= 2 min( j, j )
21 Representation of LTI Systems in the WPT Domain Cross-band filters energy
22 Representation of LTI Systems in the WPT Domain Matrix-vector notation
23 Representation of LTI Systems in the WPT Domain Matrix-vector notation Multichannel Diagonalization
24 Extension to 2D-LSI systems Cross-band filters notation Matrix-vector notation diagonal-block - DB 4 4, 4 J J J 2 ( P ) J ( ) J 2 4 4 non-zero elements out of 2 P P P P = 256, J=3 P 4 10 4 9 J 4 P 4 10 2 6
25 Layout Introduction - Wavelet-Based Image Restoration LTI Systems Representation in the WPT Domain Multiplicative Operators in the WPT Domain Image Restoration with Cross-band Filters Conclusion
26 Multiplicative Operators in the WPT Domain Cross-band filters notation cross-band filters Multirate operations between subbands Sparsity depends on the number of decomposition levels Multiplicative transfer function (MTF) operator Cross-terms are neglected Band-to-band filters are replaced with scalar multiplication
27 Multiplicative Operators in the WPT Domain Multiplicative transfer function (MTF) operator Incapable of attaining an accurate representation of LTI systems Time-varying operator, due to the downsampling on each subband
28 Multiplicative Operators in the WPT Domain Averaged MTF operator The averaged MTF operator over translations is a zero-phase LTI system with frequency-response : Shift operator
29 Multiplicative Operators in the WPT Domain LTI systems approximation using averaged MTF operator The parameters set that minimizing the least-squares frequency response error : is given by the solution of a linear equation system : where :
30 Multiplicative Operators in the WPT Domain LTI systems approximation using averaged MTF operator
31 Multiplicative Operators in the WPT Domain Cross-MTF (CMTF) operator The averaged CMTF operator over translations is an LTI system with frequency-response :
32 Multiplicative Operators in the WPT Domain LTI systems approximation using averaged CMTF operator The parameters set that minimizing the least-squares frequency response error : is given by the solution of a linear equation system : where :
33 Multiplicative Operators in the WPT Domain LTI systems approximation using averaged CMTF operator
34 Layout Introduction - Wavelet-Based Image Restoration LTI Systems Representation in the WPT Domain Multiplicative Operators in the WPT Domain Image Restoration with Cross-band Filters Conclusion
35 Image Restoration with Cross-band Filters Observation model - Additive white Gaussian noise Prior model - Laplace iid on each subband Optimization problem
36 Image Restoration with Cross-band Filters Conjugate gradient iterations where : Efficient implementation in the multichannel-dft domain
37 Image Restoration with Cross-band Filters Hyperparamaters set selection : The true image, and therefore is not available Using the MTF approximation : ˆ θ k y p, p, k k p, p, k 1 2 1 2 1 = c + λ k c x p, p 1 2 P y 2 k p, p, k 1 2 MTF approximation needs to be justified
38 Experimental Results - Setup 1 True Image Observed Image
39 Experimental Results - Setup 1
40 Experimental Results - Setup 2 True Image Observed Image
41 Experimental Results - Setup 2
42 Layout Introduction - Wavelet-Based Image Restoration LTI Systems Representation in the WPT Domain Multiplicative Operators in the WPT Domain Image Restoration with Cross-band Filters Conclusion
43 Summary LTI systems representation using cross-band filters Perfect representation Each cross-term is a multirate filter, depends on the scales ration between the frequency subbands Efficient implementation in the multichannel DFT domain Applicable for any admissible wavelet packet decomposition Multiplicative operators in the WPT domain Cross-band filtering is replaced with scalar multiplication Very low computational complexity LTI systems approximation using averaged MTF/CMTF operator Image restoration with cross-band filters Cross-band filters notation enables fast implementation MTF approximation is used for hyperparamters set selection
44 Future Research Integration of cross-band filters usage with other state-of-theart image priors Better performance at the cost of computational efficiency Image restoration in wavelet packet bases A considerable degree of freedom Finding a best basis for restoration is nontrivial Extension to time-varying systems