CS711008Z Algorithm Design and Analysis

Transcription

1 CS711008Z Algorithm Design and Analysis Lecture 5 FFT and Divide and Conquer Dongbo Bu Institute of Computing Technology Chinese Academy of Sciences, Beijing, China 1 / 56

2 Outline DFT: evaluate a polynomial at n special points; FFT: an efficient implementation of DFT; Applications of FFT: multiplying two polynomials (and multiplying two n-bits integers); time-frequency transform; solving partial differential equations; Appendix: relationship between continuous and discrete Fourier transforms 2 / 56

3 DFT: Discrete Fourier Transform DFT evaluates a polynomial A(x) = a 0 + a 1 x + + a n 1 x n 1 at n distinct points 1, ω, ω 2,, ω n 1, where ω = e 2π n i is the n-th complex root of unity Thus, it transforms the complex vector a 0, a 1,, a n 1 into another complex vector y 0, y 1,, y n 1, where y k = A(w k ), ie, y 0 = a 0 + a 1 + a 2 + a n 1 y 1 = a 0 + a 1 ω 1 + a 2 ω 2 + a n 1 ω n 1 y n 1 = a 0 + a 1 ω n 1 + a 2 ω 2(n 1) + a n 1 ω (n 1)2 Matrix form: y 0 y 1 y n 1 = ω 1 ω 2 ω n 1 1 ω 2 ω 4 ω 2(n 1) 1 ω n 1 ω 2(n 1) ω (n 1)2 a 0 a 1 a n 1 3 / 56

4 FFT: a fast way to implement DFT [Cooley and Tukey, 1965] Direct matrix-vector multiplication requires O(n 2 ) operations when using the Horner s method, ie, A(x) = a 0 + x(a 1 + x(a xa n 1 )) FFT: reduce O(n 2 ) to O(n log 2 n) using divide-and-conquer technique How does FFT achieve this? Or what calculations are redundant in the direct matrix-vector multiplication approach? Note: The idea of FFT was proposed by James Cooley and John Tukey in 1965 when analyzing earth-quake data, but the idea can be dated back to F Gauss 4 / 56

5 Let s evaluate A(x) at four special points It is easy to evaluate any 1-degree polynomial A(x) = a 0 + a 1 x at two points 1, 1 Now let s evaluate a 3-degree polynomial A(x) = a 0 + a 1 x + a 2 x 2 + a 3 x 3 at four special points 1, i, 1, i Divide: Break the polynomial into even and odd terms, ie, A even (x) = a 0 + a 2 x A odd (x) = a 1 + a 3 x Then we have the following equations: A(x) = (a 0 + a 2 x 2 ) + x(a 1 + a 3 x 2 ) = A even (x 2 ) + xa odd (x 2 ) A( x) = (a 0 + a 2 x 2 ) x(a 1 + a 3 x 2 ) = A even (x 2 ) xa odd (x 2 ) Combine: For the 4 special points 1, i, 1, i, we have A(1) = A even (1) + A odd (1) A(i) = A even ( 1) + ia odd ( 1) A( 1) = A even (1) A odd (1) A( i) = A even ( 1) ia odd ( 1) In other words, the values of A(x) at 4 points 1, i, 1, i can be calculated based on the values of A even (x), A odd (x) at 2 points 1, 1 5 / 56

6 An example: n = 4 y 0 = a 0 + a 1 + a 2 + a 3 y 1 = a 0 + a 1 ω 1 + a 2 ω 2 + a 3 ω 3 y 2 = a 0 + a 1 ω 2 + a 2 ω 4 + a 3 ω 6 y 3 = a 0 + a 1 ω 3 + a 2 ω 6 + a 3 ω 9 Objective: Evaluate A(x) at 4 points: 1, ω, ω 2, ω 3, where ω = e 1 4 2πi I ω +i ω ω 4 R ω 3 i 6 / 56

7 Step 1: Simplification y 0 = a 0 + a 1 + a 2 + a 3 y 1 = a 0 + a 1 ω 1 + a 2 ω 2 + a 3 ω 3 y 2 = a 0 + a 1 ω 2 + a 2 + a 3 ω 2 y 3 = a 0 + a 1 ω 3 + a 2 ω 2 + a 3 ω 1 7 / 56

8 Step 2 Divide into odd- and even-terms y 0 = a 0 + a 2 + a 1 + a 3 y 1 = a 0 + a 2 ω 2 + a 1 ω 1 + a 3 ω 3 y 2 = a 0 + a 2 + a 1 ω 2 + a 3 ω 2 y 3 = a 0 + a 2 ω 2 + a 1 ω 3 + a 3 ω 1 8 / 56

9 Key observation: redundant calculations y 0 = a 0 + a 2 + a 1 + a 3 y 1 = a 0 + a 2 ω 2 + a 1 ω 1 + a 3 ω 3 y 2 = a 0 + a 2 + a 1 ω 2 + a 3 ω 2 y 3 = a 0 + a 2 ω 2 + a 1 ω 3 + a 3 ω 1 Note that the calculations in the two red frames are identical, and the calculations in the blue frames are also identical after multiplying by ω 2 Here ω 2 = 1 as ω = e 2π 4 i 9 / 56

10 Step 3: Divide-and-conquer y 0 = a 0 + a 2 + a 1 + a 3 y 1 = a 0 + a 2 ω 2 + a 1 ω 1 + a 3 ω 3 y 2 = a 0 + a 2 + a 1 ω 2 + a 3 ω 2 y 3 = a 0 + a 2 ω 2 + a 1 ω 3 + a 3 ω 1 Thus the calculations in the top-left and bottom-right frames are redundant We need only = 4 log 2 4 calculations 10 / 56

11 Another example: n = 8 y 0 = a 0 + a 1 + a 2 + a 3 + a 4 + a 5 + a 6 + a 7 y 1 = a 0 + a 1ω 1 + a 2ω 2 + a 3ω 3 + a 4ω 4 + a 5ω 5 + a 6ω 6 + a 7ω 7 y 2 = a 0 + a 1ω 2 + a 2ω 4 + a 3ω 6 + a 4ω 8 + a 5ω 10 + a 6ω 12 + a 7ω 14 y 3 = a 0 + a 1ω 3 + a 2ω 6 + a 3ω 9 + a 4ω 12 + a 5ω 15 + a 6ω 18 + a 7ω 21 y 4 = a 0 + a 1ω 4 + a 2 + a 3ω 12 + a 4ω 16 + a 5ω 20 + a 6ω 24 + a 7ω 28 y 5 = a 0 + a 1ω 5 + a 2ω 10 + a 3ω 15 + a 4ω 20 + a 5ω 25 + a 6ω 30 + a 7ω 35 y 6 = a 0 + a 1ω 6 + a 2ω 12 + a 3ω 18 + a 4ω 24 + a 5ω 30 + a 6ω 36 + a 7ω 42 y 7 = a 0 + a 1ω 7 + a 2ω 14 + a 3ω 21 + a 4ω 28 + a 5ω 35 + a 6ω 42 + a 7ω 49 Objective: Evaluate A(x) at 8 points: 1, ω, ω 2,, ω 7, where ω = e 1 8 2πi ω 4 1 ω 3 ω 5 I ω 2 +i ω 1 1 ω 7 ω8 R 11 / 56

12 Step 1: Simplification y 0 = a 0 + a 1 + a 2 + a 3 + a 4 + a 5 + a 6 + a 7 y 1 = a 0 + a 1 ω 1 + a 2 ω 2 + a 3 ω 3 + a 4 ω 4 + a 5 ω 5 + a 6 ω 6 + a 7 ω 7 y 2 = a 0 + a 1 ω 2 + a 2 ω 4 + a 3 ω 6 + a 4 + a 5 ω 2 + a 6 ω 4 + a 7 ω 6 y 3 = a 0 + a 1 ω 3 + a 2 ω 6 + a 3 ω 1 + a 4 ω 4 + a 5 ω 7 + a 6 ω 2 + a 7 ω 5 y 4 = a 0 + a 1 ω 4 + a 2 ω 8 + a 3 ω 4 + a 4 + a 5 ω 4 + a 6 + a 7 ω 4 y 5 = a 0 + a 1 ω 5 + a 2 ω 2 + a 3 ω 7 + a 4 ω 4 + a 5 ω 1 + a 6 ω 6 + a 7 ω 3 y 6 = a 0 + a 1 ω 6 + a 2 ω 4 + a 3 ω 2 + a 4 + a 5 ω 6 + a 6 ω 4 + a 7 ω 2 y 7 = a 0 + a 1 ω 7 + a 2 ω 6 + a 3 ω 5 + a 4 ω 4 + a 5 ω 3 + a 6 ω 2 + a 7 ω 1 12 / 56

13 Step 2 Divide into odd- and even-terms y 0 = a 0 + a 4 + a 2 + a 6 + a 1 + a 5 + a 3 + a 7 y 1 = a 0 + a 4 ω 4 + a 2 ω 2 + a 6 ω 6 + a 1 ω 1 + a 5 ω 5 + a 3 ω 3 + a 7 ω 7 y 2 = a 0 + a 4 + a 2 ω 4 + a 6 ω 4 + a 1 ω 2 + a 5 ω 2 + a 3 ω 6 + a 7 ω 6 y 3 = a 0 + a 4 ω 4 + a 2 ω 6 + a 6 ω 2 + a 1 ω 3 + a 5 ω 7 + a 3 ω 1 + a 7 ω 5 y 4 = a 0 + a 4 + a 2 + a 6 + a 1 ω 4 + a 5 ω 4 + a 3 ω 4 + a 7 ω 4 y 5 = a 0 + a 4 ω 4 + a 2 ω 2 + a 6 ω 6 + a 1 ω 5 + a 5 ω 1 + a 3 ω 7 + a 7 ω 3 y 6 = a 0 + a 4 + a 2 ω 4 + a 6 ω 4 + a 1 ω 6 + a 5 ω 6 + a 3 ω 2 + a 7 ω 2 y 7 = a 0 + a 4 ω 4 + a 2 ω 6 + a 6 ω 2 + a 1 ω 7 + a 5 ω 3 + a 3 ω 5 + a 7 ω 1 The specific order of these terms will be explained later 13 / 56

14 Key observation: redundant calculations y 0 = a 0 + a 4 + a 2 + a 6 + a 1 + a 5 + a 3 + a 7 y 1 = a 0 + a 4ω 4 + a 2ω 2 + a 6ω 6 + a 1ω 1 + a 5ω 5 + a 3ω 3 + a 7ω 7 y 2 = a 0 + a 4 + a 2ω 4 + a 6ω 4 + a 1ω 2 + a 5ω 2 + a 3ω 6 + a 7ω 6 y 3 = a 0 + a 4ω 4 + a 2ω 6 + a 6ω 2 + a 1ω 3 + a 5ω 7 + a 3ω 1 + a 7ω 5 y 4 = a 0 + a 4 + a 2 + a 6 + a 1ω 4 + a 5ω 4 + a 3ω 4 + a 7ω 4 y 5 = a 0 + a 4ω 4 + a 2ω 2 + a 6ω 6 + a 1ω 5 + a 5ω 1 + a 3ω 7 + a 7ω 3 y 6 = a 0 + a 4 + a 2ω 4 + a 6ω 4 + a 1ω 6 + a 5ω 6 + a 3ω 2 + a 7ω 2 y 7 = a 0 + a 4ω 4 + a 2ω 6 + a 6ω 2 + a 1ω 7 + a 5ω 3 + a 3ω 5 + a 7ω 1 Note that the calculations in the two red frames are identical, and the calculations in the blue frames are also identical after multiplying by ω 4 Here ω 4 = 1 as ω = e 2π 8 i 14 / 56

15 Step 3: Divide-and-conquer y 0 = a 0 + a 4 + a 2 + a 6 + a 1 + a 5 + a 3 + a 7 y 1 = a 0 + a 4ω 4 + a 2ω 2 + a 6ω 6 + a 1ω 1 + a 5ω 5 + a 3ω 3 + a 7ω 7 y 2 = a 0 + a 4 + a 2ω 4 + a 6ω 4 + a 1ω 2 + a 5ω 2 + a 3ω 6 + a 7ω 6 y 3 = a 0 + a 4ω 4 + a 2ω 6 + a 6ω 2 + a 1ω 3 + a 5ω 7 + a 3ω 1 + a 7ω 5 y 4 = a 0 + a 4 + a 2 + a 6 + a 1ω 4 + a 5ω 4 + a 3ω 4 + a 7ω 4 y 5 = a 0 + a 4ω 4 + a 2ω 2 + a 6ω 6 + a 1ω 5 + a 5ω 1 + a 3ω 7 + a 7ω 3 y 6 = a 0 + a 4 + a 2ω 4 + a 6ω 4 + a 1ω 6 + a 5ω 6 + a 3ω 2 + a 7ω 2 y 7 = a 0 + a 4ω 4 + a 2ω 6 + a 6ω 2 + a 1ω 7 + a 5ω 3 + a 3ω 5 + a 7ω 1 Thus the calculations in the top-left and bottom-right frames are redundant 15 / 56

16 Step 3: Divide-and-conquer y 0 = a 0 + a 4 + a 2 + a 6 + a 1 + a 5 + a 3 + a 7 y 1 = a 0 + a 4ω 4 + a 2ω 2 + a 6ω 6 + a 1ω 1 + a 5ω 5 + a 3ω 3 + a 7ω 7 y 2 = a 0 + a 4 + a 2ω 4 + a 6ω 4 + a 1ω 2 + a 5ω 2 + a 3ω 6 + a 7ω 6 y 3 = a 0 + a 4ω 4 + a 2ω 6 + a 6ω 2 + a 1ω 3 + a 5ω 7 + a 3ω 1 + a 7ω 5 y 4 = a 0 + a 4 + a 2 + a 6 + a 1ω 4 + a 5ω 4 + a 3ω 4 + a 7ω 4 y 5 = a 0 + a 4ω 4 + a 2ω 2 + a 6ω 6 + a 1ω 5 + a 5ω 1 + a 3ω 7 + a 7ω 3 y 6 = a 0 + a 4 + a 2ω 4 + a 6ω 4 + a 1ω 6 + a 5ω 6 + a 3ω 2 + a 7ω 2 y 7 = a 0 + a 4ω 4 + a 2ω 6 + a 6ω 2 + a 1ω 7 + a 5ω 3 + a 3ω 5 + a 7ω 1 Finally, we need only = 8 log 2 8 calculations 16 / 56

17 The final order 0, 1, 2, 3, 4, 5, 6, 7 0, 2, 4, 6 1, 3, 5, 7 0, 4 2, 6 3, 7 1, / 56

18 FFT Algorithm FFT(n, a 0, a 1,, a n 1 ) 1: if n == 1 then 2: return a 0 ; 3: end if 4: (E 0, E 1,, E n 2 1 ) =FFT( n 2, a 0, a 2,, a n ); 5: (O 0, O 1,, O n 2 1 ) = FFT( n 2, a 1, a 3,, a n 1 ); 6: for k = 0 to n 2 1 do 7: ω k = e 2π n ki ; 8: y k = E k + ω k O k ; 9: y n 2 +k = E k ω k O k ; 10: end for 11: return (y 0, y 1,, y n 1 ); Here FFT( n 2, a 0, a 2,, a n ) computes the polynomial A even (x) = a 0 + a 2 x + + a n x n 2 at n 2 points 1, ω 2, ω 4,, ω n 2, and FFT( n 2, a 1, a 3,, a n 1 ) computes the polynomial A odd (x) = a 1 + a 3 x + + a n 1 x n 2 at these points 18 / 56

19 Inverse Discrete Fourier Transform Inverse Discrete Fourier Transform: to determine coefficients of a polynomial a 0, a 1,, a n 1 based on n point-value pairs (1, y 0 ), (ω, y 1 ),, (ω n 1, y n 1 ), where y k = A(ω k ), and A(x) = a 0 + a 1 x + a 2 x a n 1 x n 1 Matrix form y 0 y 1 y n 1 = ω 1 ω 2 ω n 1 1 ω 2 ω 4 ω 2(n 1) 1 ω n 1 ω 2(n 1) ω (n 1)2 a 0 a 1 a n 1 It takes O(n 3 ) to calculate the inverse matrix when using the Gaussian elimination technique 19 / 56

20 Inverse Discrete Fourier Transform cont d Matrix form a 0 a 1 a n 1 = 1 n ω 1 ω 2 ω n 1 1 ω 2 ω 4 ω 2(n 1) 1 ω n 1 ω 2(n 1) ω (n 1)2 y 0 y 1 y n 1 Reason: it turns out that it is nearly its own inverse More precisely, the conjugate transpose of this matrix is its own inverse 20 / 56

21 IFFT Algorithm IFFT(n, y 0, y 1,, y n 1 ) 1: if n == 1 then 2: return y 0 ; 3: end if 4: (E 0, E 1,, E n 2 1 ) = IFFT( n 2, y 0, y 2,, y n ); 5: (O 0, O 1,, O n 2 1 ) = IFFT( n 2, y 1, y 3,, y n 1 ); 6: for k = 0 to n 2 1 do 7: ω k = e 2π n ki ; 8: a k = E k + ω k O k ; 9: a n 2 +k = E k ω k O k ; 10: end for 11: return 1 n (a 0, a 1,, a n 1 ) ; Here we assume n is the power of 2 for simplicity The normalization factors multiplying FFT and IFFT (here 1 and 1 n ) and the signs of exponents are merely conventions, and differ in some treatments 21 / 56

22 Application: fast multiplication of two polynomials (or two integers) 22 / 56

23 Multiplify two polynomials: convolution Given two polynomials A(x) = a 0 + a 1 x + a 2 x a n 1 x n 1, and B(x) = b 0 + b 1 x + b 2 x b n 1 x n 1 Let s calculate its product C(x) = A(x)B(x) = c 0 + c 1 x + c 2 x c 2n 2 x 2n 2 Brute-force (convolution): c k = k i=0 a ib k i It costs O(n 2 ) time if using the convolution technique 23 / 56

24 Conversion between two representations of polynomials An efficient conversion between these two representations is extremely useful when multiplying two polynomials a 0, a 1,, a n 1 (x 0, y 0 ),, (x n 1, y n 1 ) 24 / 56

25 Using FFT to speed up multiplication Given two polynomials A(x) = a 0 + a 1 x + a 2 x a n 1 x n 1, and B(x) = b 0 + b 1 x + b 2 x b n 1 x n 1 Let s calculate its product C(x) = A(x)B(x) = c 0 + c 1 x + c 2 x c 2n 2 x 2n 2 Brute-force: c k = k i=0 a ib k i Cost O(n 2 ) time Using FFT and IFFT: O(n log n) a 0,a 1,,a n 1 b 0,b 1,,b n 1 c 0,c 1,,c 2n 1 FFT: O(n log n) IFFT: O(n log n) A(1), A(ω),, A(ω 2n 1 ) B(1), B(ω),, B(ω 2n 1 ) Multiply: O(n) C(1), C(ω),, C(ω 2n 1 ) 25 / 56

26 An example A(x) = 1 + 2x B(x) = 3 + 4x C(x) = A(x)B(x) = c 0 + c 1 x + c 2 x 2 + c 3 x 3 x 1 i 1 i A(x) 3 1 2i i B(x) 7 3 4i i C(x) i i By running IFFT(4, (21, 5 10i, 1, i)), we obtained the coefficients as c 0 = 3, c 1 = 10, c 2 = 8, and c 3 = 0 Extension: given two n-bit integers a = a n 1 a 1 a 0, and b = b n 1 b 1 b 0, it takes O(n log n) complex arithmetic steps to calculate c = a b 26 / 56

27 Application: time-frequency transform 27 / 56

28 DFT: time-domain vs frequency-domain DFT, denoted as X = F{x}, transforms a sequence of N complex numbers x 0, x 1,, x N 1 (time-domain) into a N-periodic sequence of complex numbers X 0, X 1,, X N 1 (frequency-domain): X k = N 1 n=0 x n e 2π N ikn, k = 0, 1,, N 1 Here, X k encodes both amplitude and phase of a sinusoidal component e 2π N kni of the function x n (the sinusoid s frequency is k cycles per N samples) Inverse transform of DFT: x n = 1 N N 1 k=0 X k e 2π N ikn An interpretation of DFT is that its inverse transform is the discrete analogy of the formula for a Fourier series: f(x) = + n= F n e inx, F n = 1 2π f(x)e inx dx 28 / 56

29 Time-frequency transformation FFT transforms the input data a 0, a 1,, a n 1 (time-domain samples) into y 0, y 1,, y n 1 (frequency domain) For example, y 0 = a 0 + a 1 + a 2 + a 3 + a 4 + a 5 + a 6 + a 7 y 1 = a 0 + a 1ω 1 + a 2ω 2 + a 3ω 3 + a 4ω 4 + a 5ω 5 + a 6ω 6 + a 7ω 7 y 2 = a 0 + a 1ω 2 + a 2ω 4 + a 3ω 6 + a 4ω 8 + a 5ω 10 + a 6ω 12 + a 7ω 14 y 3 = a 0 + a 1ω 3 + a 2ω 6 + a 3ω 9 + a 4ω 12 + a 5ω 15 + a 6ω 18 + a 7ω 21 y 4 = a 0 + a 1ω 4 + a 2 + a 3ω 12 + a 4ω 16 + a 5ω 20 + a 6ω 24 + a 7ω 28 y 5 = a 0 + a 1ω 5 + a 2ω 10 + a 3ω 15 + a 4ω 20 + a 5ω 25 + a 6ω 30 + a 7ω 35 y 6 = a 0 + a 1ω 6 + a 2ω 12 + a 3ω 18 + a 4ω 24 + a 5ω 30 + a 6ω 36 + a 7ω 42 y 7 = a 0 + a 1ω 7 + a 2ω 14 + a 3ω 21 + a 4ω 28 + a 5ω 35 + a 6ω 42 + a 7ω 49 Here y k encodes both amplitude and phase of a sinusoidal component of the time-domain samples a 0, a 1,, a 7 29 / 56

30 FFT: an example N = 8; t = 0:1/N:1-1/N; a = 1*cos(2*pi*1*t) + 2*sin(2*pi*3*t); Freq = 0:N-1; bar( Freq, abs(fft(a)), "b", 02 ); 30 / 56

31 y k encodes amplitude and phase of a sinusoidal component y k = a 0 + a 1 ω k + a 2 ω 2k + + a 7 ω 7k computes the dot product of two vectors: the time-domain samples a 0, a 1,, a 7, and a sinusoid signal 1, ω k, ω 2k,, ω 7k Here ω = e 2π 8 i and thus the sinusoid has frequency of k cycles per 8 samples The dot product y k = 0 if the time-domain samples a 0, a 1,, a 7 do not consist of any sinusoidal component of such frequency The reason is that: The dot product of these two vectors y k = 7 j=0 a jω jk is essentially a discrete analogy of the integral of two sinusoids, say 2π cos mx cos nxdx 0 The orthogonality of sinusoids states that for two integers m, n, 2π 0 2π sin mx sin nxdx = 0, 0 cos mx sin nxdx = 0, and 2π 0 cos mx cos nxdx = 0 (m n) 31 / 56

32 Calculation of y 1 1, ω 1, ω 2,, ω 7 represents a sinusoidal signal of frequency 1 cycle per 8 samples, and the existence of such sinusoidal component in the time-domain samples a 0, a 1,, a 7 is encoded by the dot product y 1 = a 0 + a 1 ω 1 + a 2 ω a 7 ω 7 a = [ ] c = [ ] s = [ ] sqrt( (a*c') 2 + (a*s') 2 ) = 4 32 / 56

33 Calculation of y 2 1, ω 2, ω 4,, ω 14 represents a sinusoidal signal of frequency 2 cycles per 8 samples, and the existence of such sinusoidal component in time-domain samples a 0, a 1,, a 7 is encoded by the dot product y 2 = a 0 + a 1 ω 2 + a 2 ω a 7 ω 14 a = [ ] c = [ ] s = [ ] sqrt( (a*c') 2 + (a*s') 2 ) = 0 33 / 56

34 Calculation of y 3 1, ω 3, ω 6,, ω 21 represents a sinusoidal signal of frequency 3 cycles per 8 samples, and the existence of such sinusoidal component in time-domain samples a 0, a 1,, a 7 is encoded by the dot product y 3 = a 0 + a 1 ω 3 + a 2 ω a 7 ω 21 a = [ ] c = [ ] s = [ ] sqrt( (a*c') 2 + (a*s') 2 ) = 8 34 / 56

35 Appendix: Relationship between continuous and discrete Fourier transforms 35 / 56

36 Fourier series, Fourier transform, DTFT, and DFT Fourier series decomposes a periodic function into a set of sine/cosine waves, and one of the motivations of Fourier transform comes from the extension of Fourier series to non-periodic functions DTFT uses discrete-time samples of a continuous function as input, and generates a continuous function of frequency Using a finite sequence of equally-spaced samples of a function as input, DFT computes a sequence of identical length, representing equally-spaced samples of DTFT The interval at which the DTFT is sampled is reciprocal of the duration of the input sequence The inverse DFT is a Fourier series using the DTFT samples as coefficients of corresponding frequency, and it is essentially a periodic summation of the original input sequence 36 / 56

37 Fourier series: history Figure: Jean-Baptiste Joseph Fourier ( ) In 1807, Joseph Fourier proposed the idea of Fourier series when solving heat equation, a partial differential equation Prior to Fourier s work, no solution to heat equation was known in the general case However, when the heat source was a simple sine or cosine wave, solutions were known (called eigensolutions) Thus, Fourier modelled complicated heat source as a superposition of simple sine/cosine waves, and rewrote the solution as superposition of corresponding eigensolutions 37 / 56

38 Fourier series Fourier series is a way to represent a periodic function of time as the sum of a set of simple sines and cosines (or, equivalently, complex exponentials) For example, the Fourier series of a periodic function f(x) (period 2π) is: f(x) = a 0 + (a n cos nx + b n sin nx) n=1 where a n = 1 π b n = 1 π a 0 = 1 2π 2π 0 2π 0 2π 0 f(t)dt f(t) cos ntdt (n = 1, 2, ) f(t) sin ntdt (n = 1, 2, ) 38 / 56

39 Fourier series: orthogonality of basis functions Unlike Taylor s expansion, the basis functions of Fourier series are orthogonal over [0, 2π], ie, for two integers m, n, 2π 0 1 sin xdx = 0, 2π 0 1 cos xdx = 0 2π 0 sin mx sin nxdx = 0, 2π 0 2π 0 cos mx cos nxdx = 0 (m n) cos mx sin nxdx = 0 The orthogonality plays an important role in solving coefficients a 0, a n, b n 39 / 56

40 Fourier series: complex exponential form According to the Euler s formula e ix = cos x + i sin x, we have cos x = 1 2 (eix + e ix ), sin x = 1 2i (eix e ix ), and f(x) = a 0 + (a n cos nx + b n sin nx) n=1 1 = a 0 + (a n 2 (einx + e inx 1 ) + b n 2i (einx e inx )) n=1 = a 0 + ( 1 2 (a n ib n )e inx (a n + ib n )e inx ) n=1 Define F 0 = a 0, and F n = 1 2 (a n ib n ) (n > 0) We have F n = 1 2 (a n + ib n ), and thus rewrite the Fourier series as: f(x) = + n= F n e inx, F n = 1 2 (a n ib n ) = 1 2π 2π Complex exponential form is necessary as the complex coefficients F n (called frequency spectrum) could encode both amplitude and phase of basic waves 40 / 56 0 f(t)e int dt

41 Fourier series: example 1 Periodic function f(x) = { 1 2 (π x) 0 < x 2π f(x + 2π) otherwise f(x) π 2 3π 2π π 0 π 2π 3π x π 2 Fourier series: f(x) = n=1 1 n sin nx (since a n = 0, b n = 1 n ) f(x) π 2 3π 2π π 0 π 2π 3π x π 2 41 / 56

42 Fourier series: extension to f(x) with period of 2L For a periodic function f(x) with period of 2L, the Fourier series is: f(x) = a 0 + The coefficients are: (a n cos π L nx + b n sin π L nx) n=1 a 0 = 1 2L a n = 1 L b n = 1 L 2L 0 2L 0 2L 0 f(t)dt f(t) cos π L ntdt f(t) sin π L ntdt 42 / 56

43 Fourier series: example 2 Periodic function f(x) = period T = 1 { 1 x < x T, and f(x) has a 2 f(x) x Fourier series: f(x) = 1 2T + 2 π n=1 1 n sin( π 2T n) cos(2π T nx) f(x) x 43 / 56

44 Convergence of Fourier series: Dirichlet s conditions Dirichlet s theorem states the sufficient conditions for the convergence of Fourier series, ie, if f(x) satisfies the following conditions: 1 f(x) is periodic, and absolutely integrable over a period; 2 f(x) must have a finite number of maxima and minima in any bounded interval; 3 f(x) must have a finite number of discontinuities in any bounded interval, and the discontinuity cannot be infinite Then a 0 + m (a n cos nx + b n sin nx) 1 (f(x + 0) + f(x 0)) 2 n=1 when m A succinct proof using Dirac s δ function can be found in Mathematical Methods for Physics (by Q Gu) 44 / 56

45 Proof Since a n cos nx + b n sin nx = 1 π sum of Fourier series is: S m (x) = 1 2π = = π π π f(t) cos n(x t)dt, the partial m f(t)[1 + 2 cos n(x t)]dt π n=1 π π π π f(t) sin((m )(x t)) 2π sin 1 dt 2 (x t) f(t)d m (x t)dt Here D m (x) = 1 2π (1 + 2 cos x + 2 cos 2x cos mx) Note that lim m D m (x) = δ(x) since D m (0) = 1 2π (2m + 1) Thus, we have lim S m(x) = m π π π continous at x) Please refer to Mathematical Methods for Physics π D m (x)dx = 1 and f(t)δ(x t) = f(x) (when f(x) is 45 / 56

46 Fourier transform (in terms of angular frequency ω) Fourier transform of a function of time (a signal) is a complex-valued function of frequency (represented as angular frequency ω), whose absolute value represents the amount of that frequency present in the original function F(ω) = f(x)e ixω dx, f(x) = 1 2π F(ω)e iωx dω Fourier transform, denoted as F(ω) = F{f(x)}, is called frequency representation of the original signal, and F(ω) is called spectral density { 1 x For example, the Fourier transform of 1 4 f(x) = 0 otherwise is F(ω) = f(x)e iωx dx = 2 ω sin( ω 4 ) f(x) 1 F(ω) x 8π 4π 0 4π 8π ω 46 / 56

47 Fourier transform (in terms of ordinary frequency ν) For a sinusoidal wave with period T (measured in seconds), its frequency can be measured using angular frequency ω (measured in radians per second) or using ordinary frequency ν (measured in cycles per second, or hertz), where ω = 2πν, and ν = 1 T When using angular frequency ω, Fourier transform is defined as: F(ω) = f(x)e ixω dx f(x) = 1 F(ω)e iωx dω 2π Replacing ω with ω = 2πν, we obtain another representation of Fourier transform in terms of ordinary frequency ν: F(ν) = f(x) = f(x)e 2πixν dx F(ν)e 2πixν dν 47 / 56

48 Connection between Fourier series and Fourier transform For a function that are zero outside an interval, we can calculate Fourier series on any larger interval As we lengthen the interval, the coefficients of Fourier series will approach Fourier transform f(x) T = 1 1 2π Fn ω 1 ω = 2π x 8π 4π 0 4π 8π ω f(x) T = 2 1 2π Fn ω 1 ω = π x 8π 4π 0 4π 8π ω f(x) 1 T = F(ω) x 8π 4π 0 4π 8π ω 48 / 56

49 Fourier transform: deduction Consider a periodic function f(x) with period 2L Its Fourier series f(x) = a 0 + n=1 (a n cos π L nx + b n sin π Lnx) can be rewritten as f(x) = a 0 + n=1 (a n cos ω n x + b n sin ω n x), where ω n = π Ln represents angular frequency Intuitively, when L, f(x) becomes a non-periodic function over (, ), and ω dω n=1 In particular, we have a 0 = 1 absolutely integrable, and a n cos ω nx = n=1 = n=1 2L n=1 0 0 L L L f(t)dt 0 since f(x) is 1 [ L f(t) cos ω ] ntdt cos ω nx L L ω [ L π dω [ 1 π L f(t) cos ω ntdt ] cos ω nx f(t) cos ωtdt ] cos ωx 49 / 56

50 Fourier transform: deduction cont d Similarly, we have b n sin ω nx n=1 and rewrite Fourier series as: f(x) = 1 π = 1 π = 1 2π = 1 2π = 1 2π = 1 2π 0 ω=0 t= ω=0 t= ω=0 t= 0 dω [ 1 f(t) sin ωtdt ] sin ωx π f(t)(cos ωx cos ωt + sin ωx sin ωt)dtdω f(t) cos ω(x t)dtdω f(t)(e iω(x t) + e iω(x t) )dtdω [ f(t)e iω(x t) dω + F(ω)dt f(t)e iω(x t) dωdt 0 f(t)e iω(x t) dω ] dt 50 / 56

51 Fourier transform: properties Linear operations performed in one domain (time or frequency) have corresponding operations in the other domain Differentiation in time domain corresponds to multiplication in the frequency domain, usually making it easier to analyze Convolution in the time domain corresponds to the ordinary multiplication the frequency domain Functions that are localized in one domain have Fourier transforms that are spread out across the other domain, known as the uncertainty principle The Fourier transform of a Gaussian function is another Gaussian function 51 / 56

52 Fourier transform: Poisson summation formula For a function f(x) with its Fourier transform (in terms of ordinary frequency) F(ν) = f(x)e 2πixν dx, the Poisson summation formula states k= f(k) = k= F(k) { 1 x For example, the Fourier transform of 4 f(x) = 0 otherwise is F(ν) = f(x)e 2πiνx dx = 1 πν sin( πν 2 ) Poisson summation formula states that f(k) = 1 = F(k) k= k= f(x) 1 F(ν) x 8π 4π 0 4π 8π ω 52 / 56

53 DTFT Discrete-time Fourier transform (DTFT) refers to Fourier analysis on the uniformly-spaced samples of a continuous function, ie, a Fourier series with x n as coefficients: X(ω) = x ne inω n= Here, the frequency variable ω has normalized units of radians/sample f(x) 1 F(ω) x 8π 4π 0 4π 8π ω x n 1 X(ω) x 8π 4π 0 4π 8π ω 53 / 56

54 Inverse transform of DTFT DTFT is itself a periodic function of frequency X(ω) From this function, the original samples x n can be readily recovered as below: x n = 1 X(ω)e inω dω 2π For example, the DTFT is X(ω) = cos ω + 2 cos 2ω The original samples can be recovered as: x 0 = 1 2π (1 + 2 cos ω + 2 cos 2ω)dω = 1 Similarly, we obtained x 1 = x 1 = x 2 = x 2 = 1 x n 1 X(ω) x 8π 4π 0 4π 8π ω 54 / 56

55 DTFT and DFT From these samples, DTFT produces a function of frequency that is a periodic summation of the Fourier transform of the original continuous function The sampling theorem states the theoretical conditions under which the original function can be perfectly recovered from DTFT of the samples When the input data sequence x n is N-periodic, DTFT reduces to DFT, ie, N 1 X k = x ne 2π N kni n=0 Alternatively, DTFT is itself a continuous function, and the discrete samples of it can be efficiently calculated using DFT 55 / 56

56 Appendix: Dirac s δ function Dirac s δ function { has the following two properties: x = 0 1 δ(x) = 0 otherwise 2 + δ(x)dx = 1 We can prove the following properties: For any contineous function f(x), + f(x)δ(x x 0 )dx = f(x 0 ) δ(x) is the Fourier transform of 1 since F{δ(x)} = + δ(x)e ixω dx = 1 According to the inverse Fourier transform of 1, we have: δ(x) = 1 2π e iωx dω = 1 2π e iωx dω 56 / 56