Improved Interpolation

Size: px

Start display at page:

Download "Improved Interpolation"

Scot Franklin
5 years ago
Views:

1 Improved Interpolation Interpolation plays an important role for motion compensation with improved fractional pixel accuracy. The more precise interpolated we get, the smaller our prediction residual becomes, and the more compression we get. How do we get improved and practical interpolation? In Computer Graphics we have an analog problem of interpolating samples, but there it is samples given by a model or a designer, instead of the pixel grid of our image. Hence it makes sense to take a look at the approaches in Computer Graphics: Predetermined points (e.g. our samples) interpolated curve with various desired properties Parallel to Nyquist-Sampling Theorem for

2 reconstruction: -Interpolation of samples. -Tells us how to do the interpolation: With an ideal low pass filter, whose upper cutoff frequency is half the sampling rate (for an interpolation or upsampling factor of 2). The samples are our control points (the samples in above picture). Nyquist Reconstruction (or interpolation): Our samples correspond to our control points (with zeros in between). The result of their low pass filterings is our interpolation. Interpolation Samples Nyquist resonstruction assures that there are no frequency components above the Nyquist frequency, hence minimum frequency of change, maximally smooth interpolated curve. The sinc function

3 sinc(t)= sin(π t) π t is the impulse response of our ideal low pass filter as interpolation function: Low pass fltering: Convolution with this function Zero crossings in integers We assume that our samples are on integer sample points. The interpolation function of the other samples have a zero there, hence the interpolated curve has no influence of the other samples there, and the curve will go through the sample values. The original sampling points are maintained after the interpolation,

4 which is what we want. This can be seen in the following. In continuous space or time we can write our sequence of samples as sequence of Dirac pulse multiplied by the sample values, x(n)δ(t n). n=0 Filtering is a convolution of our samples with the Sinc function. Continuous time convolution is (e.g. with a(t) our sample sequence and b(t) our ideal low pass filter impulse response) Nyquist Samples/Kontrollpunkte Delta function - (Kronecker delta, integral over Kronecker delta is 1)

5 Result Parts will be summed Sinc function is infnite polynomial, infnitely continuously diferentiable

6 interpolated curve is infnitely diferentiable! Better than, for example Cubic splines (Computer Graphics), which are continuously diferentiable only until the nnd derivative. Problem: sinc function is infnitely long, not so practical. Alternative options for interpolation flter: convolution of our samples with the so-called "Box function" h 1 (t) as a basis function. t Value is retained until the next one comes. Resultat: No continuity, problem because we have mostly smooth intensity gradients in images. Another way to preserve the continuity: Linear interpolation:

7 Basis function is the Box function convolved with itself. h 2 (x) t = = t * h 2 (t)=h 1 (t) h 1 (t) Interpolation with this basis function t Q_1(u) Q_2(u) Sum Linear interpolation f(0) 0<u<1 f(1) Observe: After doing the convolution, we obtain an increasing and a decreasing part for each to be interpolated interval, as can be seen in above picture. Each interval between two sampling points contains two functions, the rising and the falling slope of our basis function. t

8 We can write this basis function as two linear functions or polynomials of 1 st order of a local variable u, which varies between 0 and 1 in this interval between two neighboring sampling points. For the rising part we get, Q 1 (u)=u for the falling part we get, Q 2 (u)=1 u Let the sample value at the lower end of the interval be f (0) and at the upper end be f (1). Then we can linearly interpolate the in-between value at point u as f (u)=f (0) Q 2 (u)+f (1) Q 1 (u) or f (u)=f (0) (1 u)+u f (1) Observe: This sum results from the convolution of the sampling points with the basis function. Example: We have n neighboring pixels, f (0)=10, f (1)=12. What is the sample right in the middle, using linear interpolation? Answer: We use our interpolation formula with u=0.5, f (u)=f (0) (1 u)+u f (1)=10 (1 u)+u 12= = =11

9 Observe: The convolution in space or time is a multiplication in the frequency domain. Hence a convolution of two box functions becomes a squared sinc function in the frequency domain! Note: basic function is 0 for the other samples Interpolated curve goes through our samples. C0 continuity, but no C1 continuity of the frst derivative. Next step: h 4 (t )=h 2 (t ) h 2 (t )=h 1 (t) h 1 (t ) h 1 (t ) h 1 (t) cubic B-Spline Observe: In the frequency domain, this B-Spline is a sinc function to the 4 th power! (sinc⁴) (The B is for Basis )

10 Example: We compute the discrete B-Spline basis function using convolution with Python, by discretizing each unit interval into 10 steps: import numpy as np import matplotlib.pyplot as plt import scipy.signal as sig h1=np.ones(10) h2=sig.convolve(h1,h1) h4=sig.convolve(h2,h2) plt.plot(h4) plt.show() Q1 Q2 Q3 Q4

11 Observe: We now have 4 unit intervals (between the sample points, in the plot at multiples of 10). Each unit interval will have its own polynomial, Q 1 (u),...,q 4 (u), as depicted in above plot. Observe that in this way we get a very compact support, which means our impulse response is quite short (length only from 0 to 4), and it can be represented by polynomials Q i (u) of third order. Both makes it quite efcient for an implementation, which is important for video coding purposes. The drawback is now that the B-Spline does not go through 0 at the other samples (at positions 1 and 3), such that an interpolation with a simple convolution will not go through the original sampling points/values, which is not good. To fx this, we can pre-process the original signal (image) frst, such that the original samples are again obtained after the convolution with the B- Splines, as we will see. We start with the 4 th derivative, because it is easy to compute, and then we integrate 4 times,

12 to obtain the polynomial functions Q_i itself. We can use this convolution for our interpolation, but this only produces interpolated values on a fxed grid between the sample points. If we want to have an interpolated value at an arbitrary position u between n sample points, we have to fnd the polynomials Q 1 (u),...,q 4 (u). How do we get them? By observing the derivatives! Derivation rule: because d dt (h 1(t) h 1 (t))= d dt h 1(t) h 1 (t) d dt a(u) b(t u)du= a(u) d dt b(t u)du =a(t) d dt b(t)

13 4th Derivation of the B-splines: d 4 dt 4 h 4(t )= d dt h 1(t) d dt h 1(t) d dt h 1(t) d dt h 1(t) d dt h 1(t )=δ(t) δ(t 1) d4 dt 4 h 4(t )=δ(t) 4δ(t 1)+6 δ(t 2) 4 δ(t 3)+δ(t 4) (using δ(x x 0 ) δ(x x 1 )=δ(x x 0 x 1 ) ) Computation of these coefcients in Python: import numpy as np import scipy.signal as sig #the 2 diracs of the derivative of the box function: dh=np.array([1,-1]) cdh=sig.convolve(dh,dh) print("cdh= ",cdh ) ('cdh= ', array([ 1, -2, 1])) #4 derivatives of box function convolved:

14 ccdh=sig.convolve(cdh,cdh) print("ccdh= ",ccdh) ('ccdh= ', array([ 1, -4, 6, -4, 1])) We obtain the third derivative of our spline by integrating the 4 th derivative (without integration constant), d 3 dt 3 = (δ(t) 4 δ(t 1)+6 δ(t 2) 4 δ(t 3)+δ(t 4))dt This is a piecewise constant step function

15 Now we want to obtain the polynomials Q 1 (u),...,q 4 (u) with the local variable u. We start with Q 1 (u). d 3 dt Q 1(u)=1 3 The second derivative of Q 1 (u) is the integral of the third derivative, d 2 du Q 1(u)= 1du=u 2 next more integration, d du Q 1(u)= u du= u2 2, Q 1 (u)= u 2 2 du= u3 6

16 For Q 2 (u) we also need conditions for continuity for the second and frst derivatives, and the function value, where Q 1 (u) and Q 2 (u) touch, at u=1 for the frst interval with Q 1 (u) and u=0 for the second interval with Q 2 (u). d 3 dt Q 2(u)= 3 3 d 2 dt Q 2(u)= 3u+c 2 2 To determine c 2 we need the values of the second derivatives identical at the upper end of the lower interval ( u=1 ) and the lower end of the upper interval ( u=0 ), for continuity of the second derivative, d 2 du Q 1(u=1)= d 2 2 dt Q 2(u=0) 2 hence we get 1= 3 0+c 2 c 2 =1 and d 2 dt 2 Q 2(u)= 3u+1 to continue integration, with integration constant c 1, d dt Q 2(u)= 3 2 u2 +u+c 1

17 Continuity of the frst derivative, d du Q 1(u=1)= d dt Q 2(u=0) we get 1 2 =c 1 hence d dt Q 2(u)= 3 2 u2 +u+ 1 2 continue integration, Q 2 (u)= 1 2 u3 + u2 2 + u 2 +c 0 We need the contunuity of the function values, Q 1 (u=1)=q 2 (u=0) we get 1 6 =c 0 hence we get Q 2 (u)= 1 2 u3 + u2 2 + u

18 For the upper two intervals: mirror the lower polynomials by replacing u by 1 u! We do that by using an open source symbolic mathematics package called wxmaxima,

19 Q 3 (u) = Q 2 (1-u) = Calculated with Maxima Q 4 (u) = Q 1 (1-u) = Together: Q 1 (u)=1/6 u 3 ; Q 2 (u)=1/6( 3 u 3 +3u 2 +3 u+1) Q 3 (u)=1/6(3u 3 6u 2 +4) ; Q 4 (u)=1/6( u 3 +3u 2 3u+1) We now can also write our spline function h 4 (t) as

20 Q 1 (t) Q h 4 (t )={ 2 (t 1) Q 3 (t 2) Q 4 (t 3) for 0 t 1 for 1 t 2 for 2 t 3 for 3 t 4 (1) Hence our convolution of our samples f(n) with the spline function h 4 (t) for interpolation becomes f (t)= n f (n)h 4(t +2 n) Since the spline function h 4 (n) has its maximum at t=2, we shift it accordingly so that it is centered at t=n. Now we can rewrite the continuous time variable t as the sum of the lower integer index n 0 plus the foat ofset u on the unit interval, t=n 0 +u, f (n 0 +u)= n f (n)h 4 (n 0 +u+2 n) Since the spline function is zero below argument 0 and above 4, we can limit the sum to this range, f (n 0 +u)= n=n 0 1 n 0 +2 f (n)h 4 (n 0 +u+2 n) Since we can write the spline function as in eq. (4) with the four polynomials, we can rewrite the sum as

21 f (n 0 +u)=f (n 0 1) Q 4 (u)+f (n 0 ) Q 3 (u)+f (n 0 +1) Q 2 (u)+f (n 0 +2) Q 1 (u) or 3 f (n 0 +u)= f (n0 1 n) Q 4 n (u) n=0 (n) We see that we get the weighted sum of the 4 spline polynomials, each weighted by the corresponding sample value. Example: Take the sample points f ( 1)=1, f (0)=2, f (1)=3, f (2)=4. We assume the other samples values to be zero. This is shown in the following picture, f(-1) f(0) f(1) f(2) Interval with u as position of interpolation

22 The blue peaks are the original sample values, the green and red lines are the weighted and shifted B-Spline functions, the long red line is the sum of them. We want to interpolate a value at position 0 u 1 in the unit interval between sample points f(0) and f(1). We see that the sample points at both ends of this interval are connected to the peaks of the spline functions (polynomials Q 3 (u) and Q 2 (u) ), so they have the most infuence on the interpolated value, as expected. The n neighboring samples behind them still have some infuence on the interpolated value, but only via the tails of the spline function, Q 1 (u) and Q 4 (u). Hence the considered to be interpolated interval has contributions from all 4 polynomials, which appear weighted by their sample values, and add up, as in this formula, which is eq. (n) with n 0 =0 3 f (u)= n=0 f (n 1) Q 4 n (u) Plot made in Python with: import numpy as np

23 import matplotlib.pyplot as plt import scipy.signal as sig xu=np.zeros(40) xu[0:40:10]=np.arange(1,5) h1=np.ones(10) h2=sig.convolve(h1,h1)/10 h4=sig.convolve(h2,h2)/10 plt.plot(sig.convolve(xu,h4),'r') plt.plot(np.concatenate((np.zeros(20),xu))) plt.plot(1*h4,'g') plt.plot(np.concatenate((np.zeros(10),2*h4) ),'r') plt.plot(np.concatenate((np.zeros(20),3*h4)),'g') plt.plot(np.concatenate((np.zeros(30),4*h4)),'r') plt.title('spline Interpolation by Superposition') plt.show()

24 We can see that our B-Spline has the value 1/6 at sample position 1 and 3 ( Q 1 (1)=Q 2 (0)=Q 3 (1)=Q 4 (0)=1/6 ) and it has the value of 4/6 at its center at sample position n ( Q 2 (1)=Q 3 (0)=4/6 ). Hence the centered sample will be multiplied (after convolution) with the factor of 4/6, and the n neighboring samples with 1/6. This is also illustrated in the following picture 4/6 1/6 (made with Octave)

25 Here you can also see: If we have a constant sequence x(n)=k, then the interpolation on the integer grid values n are: 1/6 x(n 1)+ 4/6 x(n)+1/ 6 x(n+1)=(4/ 6+1/6+1/6) k =k So in this case the interpolation does indeed go through the original sample values! This is diferent for other examples: The interpolation of the sequence 1,n,3,4,0 with splines as an example can be seen in this picture:

26 Obtained with: import numpy as np import matplotlib.pyplot as plt import scipy.signal as sig xu=np.zeros(40) xu[0:40:10]=np.arange(1,5) h1=np.ones(10) h2=sig.convolve(h1,h1)/10 h4=sig.convolve(h2,h2)/10 plt.plot(sig.convolve(xu,h4),'r') plt.plot(np.concatenate((np.zeros(20),xu))) plt.show() (We need a shift by n intervals, because the spline function becomes maximal at t=n) Here we have an interpolated grid of 1/10 th of a pixel. We can also see that the interpolation does not go through the original pixel values, even though it is a nice and smooth interpolation. How do we obtain a spline interpolation through the original pixels? To see that, we frst need to analyse what happens with the original pixels when we interpolate with the B-Spline. Assume we already have a pre-processing of our original image, which produces the pixel

27 values c(n) out of our image pixels x(n). The question is, how do we produce the c(n) such that the original pixel values x(n) are maintained after the spline interpolation. We would like to have the original, which means we would like to have the following: x n =1/6 c n 1 4/6 c n 1/6 c n 1 Here the factor 1/6 comes from the spline value at the neighboring positions 1 and 3, and the factor 4/6 comes from the spline value at its center position n. c comes from our pre-processed or compensated image (pre-processing with some fltering operation). So here we eust look at the integer pixel positions. Observe that this condition has to hold for all inter values of n! How do we obtain c(n)? We need to apply the z-transform as a tool, because it is made for such recursive equations. Delays or advances can be formulated as multiplications with z ¹ or with z. The above equation in the z-domain is: X z =1/6 z 1 C z 4/6 C z 1/6 z C z

28 This can be re-written as: X z =C z 1/6 z 1 4/6 1/6 z and we obtain following expression for C(z): X (z) C (z)= 1/6 z 1 (1) +4/6+1/6 z Now we can see that we obtain an IIR flter for the generation of C(z) or c(n)! We can re-write the denominator polynomial of (1) as the product of n simpler polynomials using its roots. X (z) C (z)= 1/6 z 1 +4/6+1/6 z = X (z) 1/6 z(z 2 +4 z 1 +1) We can compute its roots using Octave/Matlab: roots([1,4,1]) ans = Hence we get X ( z) 1/6 z (1 z 1 z 1 )(1 z 2 z 1 ) = X (z) 1/6 z 2 (1 z 1 z 1 )(z 1 2 z 1) with z 1 = and z 2 = =1/ z 1 Observe that pole z 2 would result in an

29 instability. Observe that 1/ z 2 =z 1. We can rewrite our desired compensation flter C(z) as C (z)= 6 z 1 X ( z) (1 z 1 z 1 )(1 z 1 z) Here we obtain one causal and one anti-causal pole at z 1 = Note that we have a non-causal part z in our equation for c(n). How do we implement it? In images we can solve this problem by separating a causal and an anti-causal filter, and then apply the anti-causal flter backwards over our image, as shown in following images. The frst, causal part, is P (z)= X ( z) (1 z 1 z 1 ) For that, we flter along the causal direction, from left to rigth or from up to down, and generate the intermediate signal p(n), with an IIR flter with pole at z 1 :

30 x(n) + p(n)=x(n)+z 1 p(n-1) z 1 z ¹ z 1 p(n-1) Its transfer function in the z-domain hence becomes: P (z)= X (z)+ z 1 z 1 P(z) P (z)(1 z 1 z 1 )= X (z) P (z) X (z) = 1 1 z 1 z 1 Which is indeed our desired transfer function. We see that it has indeed a pole at position z=z 1. This now represents one half of our compensation flter. Then we apply the second part, flter the intermediate signal p(n) in the remaining anticausal direction, from right to left or from down to up, C (z)= 6 z 1 P( z) (1 z 1 z) We can implement it wit the following structure, c(n)= 6 z 1 p(n)+z 1 c(n+1)

31 p(n) + -6 z_1 c(n) z 1 z z 1 p(n+1) Its transfer function becomes (using again the z- transform): C (z)= 6 z 1 P (z)+ z 1 z C (z) C (z) (1 z 1 z)= 6 z 1 P (z) C ( z) P(z) = 6 z 1 1 z 1 z Which is indeed our desired transfer function. So the total structure is like in the following diagram,

32 x (n) p(n) + + c(n) -6z_1 z 1 z 1 Causal part z 1 z Anti-causal part This is exactly what the HHI proposal is: p(n)=x(n)+ z 1 p(n 1) c(n)=z 1 c(n+1) 6 z 1 p(n) where z_1 = -0.n6795, and where the resulting c(n) are used for the spline interpolation (p(n) is the result of the causal flter part).

33 The resulting system is here: Spline interpolation c(n) computation The block IIR flter generates the css, and the FIR flter is the spline interpolator. In-Loop processing is anti-blocking fltering and similar. (From: 28th Picture Coding Symposium, PCS2010, December 8-10, 2010, Nagoya, Japan, FRACTIONAL-SAMPLE MOTION COMPENSATION USING GENERALIZED INTERPOLATION Haricharan Lakshman, Benjamin Bross, Heiko Schwarz, and Thomas Wiegand) Pyhton Example import numpy as np import matplotlib.pyplot as plt import scipy.signal as sig

34 from scipy.signal import lflter x = np.array([0,0,1,2,3,4,0,0]) #upsampled x: xu=np.zeros(800) xu[::100]=x z1= a = np.array([1., -z1]) b = np.array([1.]) p=lflter(b, a, x,axis=0) c=-6*z1*np.fiplr([lflter([1],[1,-z1], (p[::-1]))])[0]; h1=np.ones((100)); h2=sig.convolve(h1,h1)/100; h4=sig.convolve(h2,h2)/100; print(len(h4)) cupsampled=np.zeros(100*len(c)) cupsampled[::100]=c xi=lflter(h4,1,cupsampled); plt.plot(xi,color='blue'); # defne xu plt.plot(np.hstack((np.zeros((200)),xu)),color='red')

35 plt.show() Explanation: As the signal we again take a ramp function, as in the previous example, and ist upsampled version, x = np.array([0,0,1,2,3,4,0,0]) #upsampled x: xu=np.zeros(800) xu[::100]=x we compute the signal from causal fltering as z1= a = np.array([1., -z1]) b = np.array([1.]) p=lflter(b, a, x,axis=0) We compute the anti-causal fltering by using fiplr at the input and output of our flter operation, to reverse the flter direction, c=-6*z1*np.fiplr([lflter([1],[1,-z1], (p[::-1]))])[0]; We compute our B-Spline basis function with 100 sampling points for each unit interval, h1=np.ones((100)); h2=sig.convolve(h1,h1)/100; h4=sig.convolve(h2,h2)/100; print(len(h4)) ans = 397 The length of only 397 instead of 400 samples

36 shows that this is only a more or less unprecise approximation of the B-Spline basis function, and for the reason of increasing accuracy we went to 100 samples per unit interval, instead of eust 10 in the previous example. Now we can compute the interpolated curve from the upsampled c, cupsampled=np.zeros(100*len(c)) cupsampled[::100]=c xi=lflter(h4,1,cupsampled); plt.plot(xi,color='blue'); plt.plot(np.hstack((np.zeros((200)),xu)),color='red') plt.show()

37 We can see that the interpolated curve (blue) indeed goes through the original samples (red) at full 100ss, up to the accuracy of our approximated spline function!

Lecture 10, Multirate Signal Processing Transforms as Filter Banks. Equivalent Analysis Filters of a DFT

Lecture 10, Multirate Signal Processing Transforms as Filter Banks Equivalent Analysis Filters of a DFT From the definitions in lecture 2 we know that a DFT of a block of signal x is defined as X (k)=