The O () notation. Definition: Let f(n), g(n) be functions of the natural (or real)

Transcription

1 The O () notation When analyzing the runtime of an algorithm, we want to consider the time required for large n. We also want to ignore constant factors (which often stem from tricks and do not indicate a better procedure). Definition: Let f(n), g(n) be functions of the natural (or real) numbers. We say that f(n) = O (g(n)) if there exist constants A, N such that f(n) A g(n) for all n N.

2 For example 1 + n = O (n) 9n 2 + 7n 28 = O ( n 2) = O (1) We may also write f(n) = g(n) + O (h(n)), meaning that f(n) g(n) = O (h(n)). For example (remember that n = n(n + 1)/2 = 1 2 n n) n = O ( n 3) n = O ( n 2) n = 1 2 n2 + O (n)

3 Lemma: Let f(n) = O (r(n)) and g(n) = O (r(n)). Then: f(n) + g(n) = O (r(n) + s(n)) f(n) g(n) = O (r(n) s(n)) f(n) + r(n) = O (r(n)) but: O (f(n)) O (g(n)) O (f(n) g(n))

4 Theorem: Let f be a monotonous function (i.e. f(n 1 ) f(n 2 ) whenever n 1 n 2 ). Then for every c > 0 and every a > 1 we have that In particular n c = O (a n ). (f(n)) c = O ( a f(n)) Similarly log n = O ( n d) for all d > 0. The following list is in ascending O-order : 1 log n n n n log n n 2 n n 2 n n n

5 Projection on Function Spaces Consider a vector space consisting of (complex-valued) functions. (If you like column vectors consider the discretized version with vectors of the form (f(t 1 ), f(t 2 ),..., f(t n )) obtained from function evaluations.) One can define a scalar product between functions as (f, g) = f(t)g(t) dt In the same way as with column vectors, it is now possible to define the projection of a function on a subspace spanned by basis functions b i.

6 We can use projection methods to decompose a function in the subspace into the basis elements: f(t) = c i b i (t) If the b i form an orthonormal basis, the projection formulae give us that c i = (f, b i ) = b i (t)f(t) dt These coefficients c i are called the Fourier Coefficients of f.

7 The Fourier Transformation Consider the functions sin(n t) and cos(n t) on the interval [0, 2π]. Then (for n m: 0 = = π = 2π 0 2π 0 2π 0 sin(nt) sin(mt) dt = sin(n 1 t) cos(n 2 t) dt sin(nt) sin(nt) dt = 2π 0 2π 0 cos(nt) cos(mt) dt cos(nt) cos(nt) dt Thus the function 1 π sin(nt), 1 π cos(nt) form an orthonormal basis for a subspace. Theorem: This subspace consists of all periodic functions.

8 In general, one can consider not just integral, but real multiples. In the limit the infinite sum becomes an integral and we have ( ) f(t) = A(ω) sin(ωt) dω B(ω) cos(ωt) dω with A(ω) = 1 π B(ω) = 1 π f(t) sin(ωt) dt f(t) cos(ωt) dt These functions A and B indicate the weight of the various frequencies.

9 Applications The Fourier transform has many applications in signal processing. We can use it for example to filter low frequencies:

10

11 Perceptual Coding Another application is in audio encoding. The perception of loudness depends on the frequency.

12 Also rhe human ear cannot recognize frequencies with low amplitude, if another frequency close by has high amplitude:

13 When recording the audio signal, we can remove such masked frequencies, this data reduction cannot be spotted by the ear. This principle is used for example in the MPEG standards (e.g. MP3 audio), similar techniques apply to pictures.

14 Interpretation over the complex numbers Things become more uniform if we move to the complex numbers. Here we have: sin(x) = eix e ix, cos(x) = eix +e ix and 2i 2 e a+ib = e a (cos b + i sin b) The Complex Fourier transform Using these relations, we get the Fourier transform of f as: (F(f))(ω) = g(ω) = 1 2π f(t)e iωt dt It fulfills (call f = F(g) the inverse transform of g) f(t) = (F(g))(t) = 1 2π g(ω)e iωt dω

15 Discretization On a computer we cannot take infinitely many values but have to discretize an continuous process. For every natural n and 0 m < n the numbers: e 2πim n lie equally distributed on the complex unit circle. We call these numbers Roots of Unity.

16 e 2πi3 8 Im e 2πi Re Let ω := e 2πi n. Then ω fulfills: 1. ω n = 1, ω m 1(1 m < n) n 1 2. ω jp = 0 for 1 p < n j=0 A number ( 1) fulfilling both properties is called a principal n-th root

17 of unity. If ω is a principal n-th root of unity, then ω 1 is as well. (This is all we need to know about complex numbers.) The integrals of the Fourier transform thus become sums in the Discrete Fourier transform (DFT). Let a = [a 0,..., a n 1 ]. Then F(a) = [u 0,..., u n 1 ] with: u j = n 1 a k ω jk k=0 If we define a polynomial p(x) = n 1 k=0 a kx k, we have u j = p(ω j ), the Fourier transformation thus consists of multiple polynomial evaluations.

18 Similarly we define the inverse discrete Fourier transform: F(u) = [v 0,..., v n 1 ] with v j = 1 n n 1 u k ω jk. k=0 It is easily checked that F and F are mutually inverse. Furthermore the inverse can be computed by the same algorithm, replacing ω by ω 1. If a R n, a naïve evaluation of the DFT takes O (n 2 ) operations. Our next aim will be to develop an O (n log n) algorithm.

19 Fast Fourier Transformation A particular efficient way to evaluate the discrete Fourier transform is the Fast-Fourier transform (FFT) algorithm of COOLEY and TUKEY: Let u = F(a). Then we can write u = V a with V = ω ω 2... ω n 1 1 ω 2 ω 4... ω 2(n 1) ω n 1 ω 2(n 1)... ω (n 1)(n 1) a Vandermonde matrix.

20 Let us consider the case n = 8. Then u 0 u 1 u 2 u 3 u 4 u 5 u 6 u 7 = ω ω 2 ω 3 ω 4 ω 5 ω 6 ω 7 1 ω 2 ω 4 ω 6 1 ω 2 ω 4 ω 6 1 ω 3 ω 6 ω 1 ω 4 ω 7 ω 2 ω 5 1 ω 4 1 ω 4 1 ω 4 1 ω 4 1 ω 5 ω 2 ω 7 ω 4 ω 1 ω 6 ω 3 1 ω 6 ω 4 ω 2 1 ω 6 ω 4 ω 2 1 ω 7 ω 6 ω 5 ω 4 ω 3 ω 2 ω 1 a 0 a 1 a 2 a 3 a 4 a 5 a 6 a 7

21 We swap the columns to sort by even/odd indices: [0, 2, 4, 6, 1, 3, 5, 7]: u 0 u 1 u 2 u 3 u 4 u 5 u 6 u 7 = ω 2 ω 4 ω 6 ω ω 3 ω 5 ω 7 1 ω 4 1 ω 4 ω 2 ω 6 ω 2 ω 6 1 ω 6 ω 4 ω 2 ω 3 ω 1 ω 7 ω ω 4 ω 4 ω 4 ω 4 1 ω 2 ω 4 ω 6 ω 5 ω 7 ω 1 ω 3 1 ω 4 1 ω 4 ω 6 ω 2 ω 6 ω 2 1 ω 6 ω 4 ω 2 ω 7 ω 5 ω 3 ω 1 a 0 a 2 a 4 a 6 a 1 a 3 a 5 a 7

22 We observe that we can write (ω, ω 3, ω 5, ω 7 ) = ω(1, ω 2, ω 4, ω 6 ) = ω(1, ζ, ζ 2, ζ 3 ) (ω 2, ω 6, ω 2, ω 6 ) = ω 2 (1, ω 4, 1, ω 4 ) = ω 2 (1, ζ 2, 1, ζ 2 ) (ω 3, ω, ω 7, ω 5 ) = ω 3 (1, ω 6, ω 4, ω 2 ) = ω 3 (1, ζ 3, ζ 2, ζ) where ζ = ω 2 is a primitive 4-th root of unity 4-dimensional fourier transform given by u 0 u 1 u 2 u ζ ζ 2 ζ 3 = 1 ζ 2 1 ζ 2 1 ζ 3 ζ 2 ζ and we have a a 0 a 1 a 2 a 3. In other words: We can write the transformation matrix F 8 as a product:

23 F 8 = F ω ω ω 3 F 4 F ω ω ω 3 F 4 where F 4 is the matrix for the 4-dimensional transformation.

24 We can now apply the same method recursively to the 4-dimensional transformation and so on. Eventually, this eliminates all matrix multiplication by scalar multiplication. For an n-dimensional problem we have log(n) iterations, each step of which has a cost of O (n). The total cost is thus O (n log n). The same process works for any power of 2 (filling up the data if there are fewer points). The inverse transform uses the same algorithm with ω replaced by ω 1.

25 function FFT(n,a,ω); (n a power of 2) Output: u = F(a). begin u := []; if n = 1 then u[0] := a[0]; else m := n/2; c :=FFT(m,a e,ω 2 ); d :=FFT(m,a o,ω 2 ); for j [0..m 1] do u[j] := c[j] + ω j d[j]; u[j + m] := c[j] ω j d[j]; od; fi; return u; end

26 Example: Sunspots We want to analyze the number of sunspots per year. Astronomers have counted these for about three centuries, their number is called the Wolfer number. We will use MATLAB to perform a fourier transform. If we log in remotely (from another X-windows workstation), we have to redirect graphics output: $ setenv DISPLAY mycomputer.math.colostate.edu:0 $ matlab Once in MATLAB we load the data (which is supplied as a sample) and split the list of pairs in a list of years and of counts. We also plot the curve.

27 load sunspot.dat year=sunspot(:,1); wolfer=sunspot(:,2); plot(year,wolfer) title( Sunspot Data )

28 n=length(y); We now perform a fast fourier transformation. This stores the coefficients in a list. The first coefficient (cos(0x) = 1) gives an absolut shift (zero bias) that is usually not interesting, so we eliminate it. Y = fft(wolfer); Y(1)=0; The first n list entries correspond to positive frequencies, the last n 2 2 corresponding to negative frequencies. Since our input signal is real, we need to consider only the entries corresponding to positive frequencies. (The rest will be their complex conjugates.) Since we are only interested in the intensities for the various frequencies, we only consider the (absolute values of) the first n 2 entries. Taking their square (proportional to the power per frequency) will emphasize peaks.

29 intens = abs(y(1:n/2)); power = intens.ˆ2;; To obtain the frequencies, we note that our frequency resolution will be 1 2 times the sampling frequency (Nyquist rate/shannon s theorem). Thus we create a equidistributed list of length n which goes from 1 to 2 rate. Also, since the frequencies are very small, we instead will use 2 their reciprocal values. rate=1; nyquist = 1/2; freq = (1:n/2)/(n/2)*nyquist*rate; period=1./freq;

30 Finally we plot the frequency distribution and set the view window to x [0, 40], y [0, ], which fits the plot. plot(period,power); ylabel( Power ); xlabel( Period (Years/Cycle) ); axis([ e+7]);

31