Sparse Sampling: Theory and Applications

Transcription

1 November 24, 29

2 Outline Problem Statement Signals with Finite Rate of Innovation Sampling Kernels: E-splines and -splines Sparse Sampling: the asic Set-up and Extensions The Noisy Scenario Applications ompression Image Super-resolution onclusions and Outlook

3 Problem Statement You are given a class of functions. You have a sampling device, typically, a low-pass filter. Given the measurements y n = x(t), ϕ(t/t n), you want to reconstruct x(t). x(t) h(t)=!(!t/t) y(t) T y n =<x(t),!(t/t!n)> Acquisition Device Natural questions: When is there a one-to-one mapping between x(t) and y n? What signals can be sampled and what kernels ϕ(t) can be used? What reconstruction algorithm?

4 Problem Statement Observed scene Lens Acquisition System D Array Samples The low-quality lens blurs the images. The images are under-sampled by the low resolution D array. You need a good post-processing algorithm to undo the blurring and upsample the images.

5 Signals with Finite Rate of Innovation Sampling is an ill-posed inverse problem. To make it well-posed we need to impose constraints on the signals. Typically signals are assumed to be bandlimited: x(t) = P n ynsinc(t/t n). What is so special about those signals? The signal x(t) = P n ynsinc(t/t n) is exactly specified by one parameter y n every T seconds, it has a finite number ρ = /T of degrees of freedom per unit of time. In the classical formulation, innovation is uniform. How about signals where the rate of innovation is finite but non-uniform? E.g. Piecewise sinusoidal signals (Frequency Hopping modulation). Pulse position modulation (UW). Edges in images.

6 Signals with Finite Rate of Innovation We assume that the signals considered are completely determined by a finite number of parameters per unit of time. The number ρ of degrees of freedom per unit of time is called rate of innovation. A signal with a finite ρ is called signal with finite rate of innovation (FRI). Notice: this is a parametric sparsity. FRI signals include: andlimited signals and signals belonging to shift-invariant subspaces. K-sparse discrete signals (like in ompressed Sensing). Signals with point-like innovation, (point source phenomena), piecewise sinusoidal signals (OFDM, FH), filtered Diracs (UW, Neuronal signals).

7 Examples of Signals with Finite Rate of Innovation Filtered Streams of Diracs Piecewise Polynomial Signals Piecewise Sinusoidal Signals Mondrian paintings ;-)

8 Sampling Kernels Any kernel ϕ(t) that can reproduce exponentials: X c m,nϕ(t n) = e αmt, α m = α + mλ and m =,,..., L. n This includes any composite kernel of the form φ(t) β α (t) where β α (t) = β α (t) β α (t)... β αl (t) and β αi (t) is an Exponential Spline of first order [Unser:5]. E Spline β α (t) e α t Notice: β α(t) ˆβ(ω) = eα jω jω α α can be complex. E-Spline is of compact support. E-Spline reduces to the classical polynomial spline when α =.

9 Kernels Reproducing Exponentials.9 e α t t t The E-spline of first order β α (t) reproduces the exponential e αt : X c,n β α (t n) = e αt. n In this case c,n = e αn. In general, c m,n = c m, e αmn.

10 Trigonometric E-Splines The exponent α of the E-splines can be complex. This means β α(t) can be a complex function. However if pairs of exponents are chosen to be complex conjugate then the spline stays real. Example: 8 >< β α +jω (t) β α jω (t) = >: sin ω t ω e α t sin ω (t 2) ω e α t t < t < 2 Otherwise When α = (i.e., purely imaginary exponents), the spline is called trigonometric spline.

11 t t t t Trigonometric E-Splines.5 Exponential Reproduction Scaled and shifted E splines.5 Exponential Reproduction Scaled and shifted E splines Exponential Reproduction Scaled and shifted E splines.5 Exponential Reproduction Scaled and shifted E splines Here α = ( jω, jω ) and ω =.2. P n cn,mβ α(t n) = e jmω m =,. Notice: β α (t) is a real function, but the coefficients c m,n are complex.

12 ! (t) ! 2 (t) ! (t) ! 3 (t) E-Splines and -splines When α m =, m =,,..., L. The E-spline reduce to the classical -spline and is then able to reproduce polynomials up to degree L. β (t) β (t) β 2 (t) β 3 (t)

13 E-Splines and -splines The E-spline reduces to the classical cubic -spline when α m =, m =,,..., L and L = 3. In this case it can reproduce polynomials up to degree L = c,n = (,,,,,, ) c,n = ( 3, 2,,,, 2, 3) c 2,n (8.7, 3.7,.7,.333,.7, 3.7, 8.7) c 3,n ( 24, 6,.,,., 6, 24)

14 Kernel Reproducing Exponential Any functions with rational Fourier transform: Q i ˆϕ(ω) = (jω b i ) Qm (jω am) m =,,..., L. is a generalized E-splines. This includes practical devices as common as an R circuit: + R + x(t) y(t) - -

15 Sparse Sampling: asic Set-up Assume that x(t) is the signal we have sampled and want to reconstruct and assume the sampling kernel ϕ(t) is any function that can reproduce exponentials of the form c m,n ϕ(t n) = e αmt m =,,..., L, n We want to retrieve x(t), from the samples y n = x(t), ϕ(t/t n).

16 Sparse Sampling: asic Set-up We have that (we assume T = for simplicity) s m = n c m,ny n = x(t), n c m,nϕ(t n) = x(t)eαmt dt, m =,,..., L. s m is the bilateral Laplace transform of x(t) evaluated at α m. When α m = jω m then s m = ˆx(ω m ). When α m =, the s m s are the polynomial moments of x(t).

17 Sampling Streams of Diracs Assume x(t) is a stream of K Diracs: x(t) = K k= x kδ(t t k ). We restrict α m = α + mλ m =,,..., L and L 2K. We obtain s m = x(t)eαmt dt, = K k= x ke αmt k = K k= ˆx ke λmt k = K k= ˆx kuk m, m =,,..., L.

18 The quantity The Annihilating Filter Method s m = K k= ˆx k u m k, m =,,..., L is a power-sum series. We can retrieve the locations u k and the amplitudes ˆx k with the annihilating filter method (also known as Prony s method since it was discovered by Gaspard de Prony in 795). Given the pairs {u k, ˆx k }, then t k = (ln u k )/λ and x k = ˆx k /e αt k.

19 The Annihilating Filter Method. all h m the filter with z-transform H(z) = P K i= h iz i = Q K k= ( u kz ). We have that KX KX K X X KX h m s m = i= h i s m i = i= k= K ˆx k h i u m i k = k= ˆx k u m k i= h i u i k {z } =. This filter is thus called the annihilating filter. In matrix/vector form, using the fact that h =, we obtain 2 3 s K s K 2 s h s K s K s K s h 2 s K+ = A s L s L 2 s L K h K s L Solve the above system to find the coefficients of the annihilating filter.

20 The Annihilating Filter Method 2. Given the coefficients {, h, h 2,..., h k }, we get the locations u k by finding the roots of H(z). 3. Solve the first K equations in s m = P K k= ˆx kuk m to find the amplitudes ˆx k. In matrix/vector form u u u K u K u K u K K 3 7 ˆx ˆx. ˆx K = lassic Vandermonde system. Unique solution for distinct u k s. s s. s K. () A

21 Sampling Streams of Diracs: Numerical Example t (a) Original Signal t (b) Sampling Kernel (β 7 (t)) t (c) Samples t (d) Reconstructed Signal

22 Sparse Sampling: Extensions Using variations of the annihilating filter methods other signals can be sampled such as filtered streams of Diracs, multi-dimensional signals and piecewise sinusoidal signals

23 Sampling Piecewise Sinusoidal Signals We consider signals of the type: x(t) = DX d= n= NX A d,n cos(ω d,n t + ϕ d,n )ξ d (t), where ξ d (t) = u(t t d ) u(t t d+ ) and < t <... < t d <... < t D+ <. Why is it difficult to sample them? Piecewise sinusoidal signals contain innovation in both spectral and temporal domains. They are not bandlimited. They are not sparse in time nor in a basis or a frame.

24 Sampling Piecewise Sinusoidal Signals [erentd:9] From the samples we can obtain the Laplace transform of x(t) at α m = α + mλ, m =,,..., L: s m = DX 2NX d= n= [e t d+(jω d,n +α m) e t d (jω d,n +α m) ] Ā d,n, (jω d,n + α m) where Ā d,n = A d,n e jϕ d,n. We define the polynomial DY Q(α m) = 2NY d= n= (jω d,n + α m) = JX r j αm. j j=

25 Sampling Piecewise Sinusoidal Signals [erentd:9] Multiplying both side of the equation by Q(α m) we obtain: Q(α m)s m = DX 2NX d= n= Ā d,n P(α m)[e t d+(jω d,n +α m) e t d (jω d,n +α m) ], (2) where P(α m) is a polynomial. Since α m = α + λm the right-hand side of (2) can be annihilated: Q(α m)s m h m =.

26 Sampling Piecewise Sinusoidal Signals [erentd:9] In matrix/vector form (assuming h = ), we have: s K α J K s K s α J s s K+ α J K+ s K+ s α J s s L α J L s L s α J (L K) s (L K) r... r J h r h r... h K r J...h K r K A =.

27 Sampling Piecewise Sinusoidal Signals From the coefficients r j, j =,,...J, we obtain Q(α m ). The roots of the filter H(z) and of the polynomial Q(α m ) give the locations of the switching points and the frequencies of the sine waves respectively. To solve the system we need L 4D 3 N 2 + 4D 2 N 2 + 4D 2 N + 6DN.

28 Numerical Example [sec] (a) [sec] (b) [sec] (c)

29 Robust Sparse Sampling In the presence of noise, the annihilation equation SH = is only approximately satisfied. Minimize: SH 2 under the constraint H 2 =. This is achieved by performing an SVD of S: S = UλV T. Then H is the last column of V. Notice: this is similar to Pisarenko s method in spectral estimation.

30 Robust Sparse Sampling: adzow s algorithm For small SNR use adzow s method to denoise S before applying TLS. The basic intuition behind this method is that, in the noiseless case, S is rank deficient (rank K) and Toeplitz, while in the noisy case S is full rank. Algorithm: SVD of S = UλV T. Keep the K largest diagonal coefficients of λ and set the others to zero. Reconstruct S = Uλ V T. This matrix is not Toeplitz, make it so by averaging along the diagonals. Iterate.

31 Robust Sparse Sampling Samples are corrupted by additive noise. This is a parametric estimation problem. Unbiased algorithms have a covariance matrix lower bounded by R. The proposed algorithm reaches R down to SNR of 5d.

32 Robust Sparse Sampling Noiseless Samples (N=28) Noisy Samples (N=28) Original and estimated Diracs : SNR = 5 d; original Diracs estimated Diracs t

33 Page 23 of 8 Piecewise sinusoidal signal location [seconds] Robust Sparse Sampling IEEE TRANSATIONS ON SIGNAL PROESSING Observed Standard Deviation 6 ramer!rao ound ! !2 5.3! ! Input SNR [d] Input SNR [d] 2 22 (a) (b) Fig. 8. Retrieval of the switching point of a step sine (ω = 2.23π [/sec] and t =.497 [sec]) in 28 noisy samples. (a) Scatter plot of the estimated location. (b) Standard deviation (averages over iterations) of the location retrieval compared 27 to the ramér-rao bound. 28 For Re Standard deviation for location t

34 Robust Sparse Sampling Page 24 of 8 IEEE TRANSATIONS ON SIGNAL PROESSING ! 9!2.5 [sec] (a) 2 Ground Truth Reconstructed Signal !.5 6! 7.5 [sec] (b) Fig. 9. Recovery of a truncated sine wave at SNR = 8 [d]. (a) The observed noisy samples. (b) The reconstructed signal 2 SNR= 8d, N= along with the ground truth signal (dashed) Pier 27Luigi Dragottitime piecewise sinusoidal signal (with t =.244 [sec], t 2 =.7324 [sec] and ω = 2.23π [rad/sec]) 28 given 28 noisy samples at an SNR of 8 [d]. Note that despite the small error in the estimation of the For Re

35 ompression FRI Signals can be sparsely sampled. an they also be compressed? What happens when the samples are quantized? Traditional ompression is based on complex encoders and simple decoders. New sampling theories are characterized by a linear acquisition but non-linear reconstruction.

36 ompression Signals are piecewise smooth, with α-lipschitz regular pieces. Traditional compression algorithms use the wavelet transform and compress only the large wavelet coefficients. They achieve the optimal D(R) performance: D(R) R 2α The proposed algorithm compresses and transmits only the low-pass coefficients of the wavelet transform (linear encoding), but uses FRI techniques to estimate the discontinuities in the signal from the low-pass coefficients (non-linear decoding).

37 Performance Analysis Any piecewise smooth signals can be decomposed into a piecewise polynomial and a globally smooth signal [DragottiV:3]. The low-pass coefficients are a sufficient representation of the piecewise polynomial signal, but quantization and the smooth signal act as noise and this reduces the reconstruction fidelity. We treat both contributions as additive noise and evaluate the R-bounds for this estimation problem. The quantization noise depends on the bit-rate R. This leads to a connection between R-ounds and rate-distortion analysis and leads to this performance bound: D FRI (R) R 2α.

38 Simulation Results D Semi Parametric c R 2! Linear Approx. c 2 R R (bits) original signal linear approx. compression, R = 736 bits semi parametric compression, R = 544 bits

39 Application: Image Super-Resolution Super-Resolution is a multichannel sampling problem with unknown shifts. Use moments to retrieve the shifts or the geometric transformation between images. (a)original (52 52) (b) Low-res. (64 64) (c) Super-res ( PSNR=24.2d) Forty low-resolution and shifted versions of the original. The disparity between images has a finite rate of innovation and can be retrieved. Accurate registration is achieved by retrieving the continuous moments of the Tiger from the samples. The registered images are interpolated to achieve super-resolution.

40 Application: Image Super-Resolution Image super-resolution basic building blocks Restoration Super-resolved image LR image... LR image k Set of low-resolution images Image Registration HR grid estimation

41 Application: Image Super-Resolution For each blurred image I (x, y): A pixel Pm,n in the blurred image is given by P m,n = I (x, y), ϕ(x/t n, y/t m), where ϕ(t) represents the point spread function of the lens. We assume ϕ(t) is a spline that can reproduce polynomials: X X c m,n (l,j) ϕ(x n, y m) = x l y j l =,,..., N; j =,,..., N. n m We retrieve the exact moments of I (x, y) from Pm,n: τ l,j = X X Z Z c m,n (l,j) P m,n = I (x, y)x l y j dxdy. n m Given the moments from two or more images, we estimate the geometrical transformation and register them. Notice that moments of up to order three along the x and y coordinates allows the estimation of an affine transformation.

42 Application: Image Super-Resolution Line spread function Pixels (perpendicular) (a)original (24 339) (b) Point Spread function

43 Application: Image Super-Resolution (a)original (28 28) (b) Super-res (24 24)

44 Application: Image Super-Resolution (a)original (48 48) (b) Super-res (48 48)

45 onclusions Sampling signals at their rate of innovation: New framework that allows the sampling and reconstruction of signals at a rate smaller than Nyquist rate. Robust and fast algorithms with complexity proportional to the number of degrees of freedom. Provable optimality (i.e. R achieved over wide SNR ranges). Intriguing connections with multi-resolution analysis, Fourier theory and analogue circuit theory. ut also There is no such thing as a free lunch. ore application is difficult. Still many open questions from theory to practice.

46 References On sampling T. lu, P.L. Dragotti, M. Vetterli, P. Marziliano and L. oulot Sparse Sampling of Signal Innovations: Theory, Algorithms and Performance ounds, IEEE Signal Processing Magazine, vol. 25(2), pp. 3-4, March 28 P.L. Dragotti, M. Vetterli and T. lu, Sampling Moments and Reconstructing Signals of Finite Rate of Innovation: Shannon meets Strang-Fix, IEEE Trans. on Signal Processing, vol.55 (5), pp , May 27. J.erent and P.L. Dragotti, and T. lu, Sampling Piecewise Sinusoidal Signals with Finite Rate of Innovation Methods, accepted for publication in IEEE Transactions on Signal Processing, July 29 On Image Super-Resolution L. aboulaz and P.L. Dragotti, Exact Feature Extraction using Finite Rate of Innovation Principles with an Application to Image Super-Resolution, IEEE Trans. on Image Processing, vol.8(2), pp , February 29. On compression V. haisinthop and P.L. Dragotti, Semi-Parametric ompression of Piecewise-Smooth Functions, in Proc. of European onference on Signal Processing (EUSIPO), Glasgow, UK, August 29.