Design and Analysis of Algorithms

Transcription

1 Design and Analysis of Algorithms CSE 5311 Lecture 5 Divide and Conquer: Fast Fourier Transform Junzhou Huang, Ph.D. Department of Computer Science and Engineering CSE5311 Design and Analysis of Algorithms 1

2 Reviewing: Master Theorem The master method applies to recurrences of the form T(n = a T(n/b + f (n, where constants a 1, b > 1, and f is asymptotically positive function 1. f (n = O(n log ba ε for some constant ε >, then T(n = Θ(n log ba 2. f (n = O(n log ba for some constant ε >, then T(n = Θ(n log ba lgn 3. f (n = O(n log ba + ε for some constant ε >, and if a f (n/b c f (n for some constant c < 1, then T(n = Θ( f (n. How to theoretically prove it? CSE5311 Design and Analysis of Algorithms 2

3 Fast Fourier Transform Applications Optics, acoustics, quantum physics, telecommunications, control systems Signal processing, speech recognition, data compression, image processing Machine learning, data mining, computer vision, big data analytics DVD, JPEG, MP3, MRI, CAT scan Charles van Loan: The FFT is one of the truly great computational developments of this [2th] century. It has changed the face of science and engineering so much that it is not an exaggeration to say that life as we know it would be very different without the FFT. CSE5311 Design and Analysis of Algorithms 3

4 Fast Fourier Transform History Gauss (185, Analyzed periodic motion of asteroid Ceres. Runge-König (1924. Laid theoretical groundwork. Danielson-Lanczos (1942. Efficient algorithm. Cooley-Tukey (1965. Monitoring nuclear tests in Soviet Union and tracking submarines. Rediscovered and popularized FFT. Importance not fully realized until advent of digital computers. CSE5311 Design and Analysis of Algorithms 4

5 Representation of Polynomials A polynomial in the variable x over an algebraic field F is representation of a function A(x as a formal sum Coefficient representation A( x n 1 = a j x j= a = ( a, a1,... an 1 j Point-value representation {( x, y,( x, y,...,( x, y } 1 1 n 1 n 1 Adding Multiplication Coefficient representation Θ( n 2 Θ( n Point-value representation Θ( n Θ( n CSE5311 Design and Analysis of Algorithms 5

6 Polynomials: Coefficient Representation Polynomial. [coefficient representation] Add: O(n arithmetic operations Evaluate: O(n using Horner's method. Multiply (convolve: O(n 2 using brute force. CSE5311 Design and Analysis of Algorithms 6

7 Polynomials: Point-Value Representation Fundamental theorem of algebra. [Gauss, PhD thesis]: A degree n polynomial with complex coefficients has n complex roots. Corollary. A degree n-1 polynomial A(x is uniquely specified by its evaluation at n distinct values of x. CSE5311 Design and Analysis of Algorithms 7

8 Polynomials: Point-Value Representation Polynomial. [Point-value representation] Add: O(n arithmetic operations Multiple: O(n, extend A(x and B(x to 2n-1 points Evaluate: O(n 2 using Lagrange's formula CSE5311 Design and Analysis of Algorithms 8

9 Converting Between Two Polynomial Representations Tradeoff between fast evaluation or fast multiplication. We want both! Goal. Make all ops fast by efficiently converting between two representations. CSE5311 Design and Analysis of Algorithms 9

10 Converting Between Two Polynomial Representations Coefficient to point-value: Given a polynomial a + a 1 x a n-1 x n-1, evaluate it at n distinct points x,..., x n-1. Brute Force! O(n 2 for matrix-vector multiply O(n 3 for Gaussian elimination Vandermonde matrix is invertible iff x i distinct Point-value to coefficient: Given n distinct points x,..., x n-1 and values y,..., y n-1, find unique polynomial a + a 1 x a n-1 x n-1 that has given values at given points. CSE5311 Design and Analysis of Algorithms 1

11 Coefficient to Point-Value Representation: Intuition Coefficient to point-value: Given a polynomial a + a 1 x a n-1 x n-1, evaluate it at n distinct points x,..., x n-1. Divide. Break polynomial up into even and odd powers. Why? Useful Trick Intuition. Choose four points to be ±1, ±i. Can evaluate polynomial of degree n at 4 points by evaluating two polynomials of degree n/2 at 2 points. CSE5311 Design and Analysis of Algorithms 11

12 Useful Trick A(x = a + a 1 x + a 2 x 2 + a 3 x a n-1 x n-1 A even (x = a + a 2 x + a 4 x a n-2 x (n-2/2 A odd (x = a 1 + a 3 x + a 5 x a n-1 x (n-2/2 Show: A(x = A even (x 2 + x A odd (x 2 CSE5311 Design and Analysis of Algorithms 12

13 Fast Multiplication Question. Can we use the linear-time multiplication method for polynomials in pointvalue form to expedite polynomial multiplication in coefficient form? Answer. Yes, but we are to be able to convert quickly from one form to another. a, a,..., a 1 n 1 b, b,..., b 1 n 1 Ordinary multiplication Time Θ(n² c,c 1,...,c2n 2 Evaluation Time Θ(n lg n Interpolation Time Θ(n lg n A ( ω, B( ω 2 n 2 n A ( ω, B( ω n 2 n M A ( ω, B( ω 2 n 1 2 n 1 2 n 2 n Pointwise multiplication Time Θ(n C( ω 2 n C( ω M 1 2 n C( ω 2 n 1 2 n CSE5311 Design and Analysis of Algorithms 13

14 Complex Roots of Unity n Z 1= There are exactly n complex roots of unity. They form a cyclic multiplication group: 2 = ik n ω k e π The value = 2 i n ω is called the primitive root of unity; all of the other complex 1 e π roots are powers of it. 2 i/8 ω = e π is the 8 th root of unity CSE5311 Design and Analysis of Algorithms 14

15 Complex Analysis Polar coordinates: re θi e θi = cos θ + i sin θ a is an n th root of unity if a n = 1 Square roots of unity: +1, -1 Fourth roots of unity: +1, -1, i, -i Eighth roots of unity: +1, -1, i, -i, β + iβ, β - iβ, -β + iβ, -β - iβ where β = sqrt(2 CSE5311 Design and Analysis of Algorithms 15

16 e 2πki/n e 2πi = 1 e πi = -1 n th roots of unity: e 2πki/n for k = n-1 Notation: ω k,n = e 2πki/n Interesting fact: 1 + ω k,n + ω 2 k,n + ω 3 k,n ω n-1 k,n = for k!= CSE5311 Design and Analysis of Algorithms 16

17 Discrete Fourier Transform (DFT Coefficient to point-value: Let F(x be the polynomial with degreebound n (power of 2, n 1 n 2 where F( x = a x + a x a n 1 n 2 Key idea: choose x k = ω k where ω is principal n th root of unity. Let y = F( ω k. Then k y L 1 a 2 n 1 y 1 ω ω ω 1 L a ( n 1 y 1 ω ω ω 2 = L * a 2 M M M M M M 2 n 1 2( n 1 ( n 1 y n 1 1 ω ω ω a L n 1 The vector y= ( y is called the Discrete Fourier Transform of vector, y1,... yn 1 a. The matrix is denoted by F ( n ω. CSE5311 Design and Analysis of Algorithms 17

18 How to find F n -1? Proposition. Let ω be a primitive l-th root of unity over a field L. Then 1 l k = if l > 1 k ω = 1 otherwise Proof. The l =1 case is immediate since ω=1. Since ω is a primitive l-th root, each ω k, k is a distinct l-th root of unity. Z Z Z Z Z l 2 l 1 1 = ( ωl ( ωl ( ωl...( ω l = l 1 l 1 l k l 1 l k ( l Z... ( 1 l k= k= = Z ω + + ω Comparing the coefficients of Z l-1 on the left and right hand sides of this equation proves the proposition. CSE5311 Design and Analysis of Algorithms 18

19 Inverse Matrix to F n Proposition. Let ω be an n-th root of unity. Then, F ( ω F ( ω = ne 1 n n n Proof. The ij element of F ( ωf ( ω th 1 n n n-1 n-1 ik ik k(i j k= k= is, if i j ω ω = ω = n, otherwise i j The i=j case is obvious. If i j then ω will be a primitive root of unity of order l, where l n. Applying the previous proposition completes the proof. So, F n ( ω = F n ( ω n Evaluating y= F n (ω a Interpolation 1 1 a= F n ( ω y n CSE5311 Design and Analysis of Algorithms 19

20 Fast Fourier Transform Goal. Evaluate a degree n-1 polynomial A(x = a + a 1 x + a 2 x 2 + a 3 x a n-1 x n-1 at its n th roots of unity: ω, ω 1,, ω n-1. Divide. Break polynomial up into even and odd powers. Conquer. Evaluate degree A even (x and A odd (x at the (n/2-th roots of unity: ν, ν 1,, ν n/2-1. Combine. CSE5311 Design and Analysis of Algorithms 2

21 Recursive FFT 1 n l e n g t h [ a ] 2 i f n = 1 3 t h e n r e t u r n a 4 ω e 5 ω 1 n 2 π i n [ ] 6 a ( a, a 2,..., a n - 2 [ 1 ] 7 a ( a 1, a 3,..., a n - 1 [ ] [ ] 8 y R e c u r s i v e - F F T ( a [ 1 ] [ 1 ] 9 y R e c u r s i v e - F F T ( a 1 f o r k t o n / d o y y + ω y [ ] [ 1 ] k k k 1 2 y y - ω y 1 3 ω ω ω 1 4 r e t u r n y [ ] [ 1 ] k + ( n / 2 k k n CSE5311 Design and Analysis of Algorithms 21

22 Time of the Recursive-FFT To determine the running time of procedure Recursive-FFT, we note, that exclusive of the recursive calls, each invocation takes time Θ(n, where n is the length of the input vector.the recurrence for the running time is therefore T(n = 2T(n/2 + Θ(n =? Θ(n log n CSE5311 Design and Analysis of Algorithms 22

23 More Effective Implementations [1] The for loop involves computing the value ω k nyk twice.we can change the loop(the butterfly operation: [] y k ω k n. for k [1] k [] y y +t k to n/2-1 do t ωy k [] y y -t k+(n/2 ω ωω [] y [] [1] k y ω k k nyk n y k +ω k y [] [1] k n k There are 1 complex multiplication and 2 complex additions CSE5311 Design and Analysis of Algorithms 23

24 x ( r 1 x ( r 2 x 1( = x( x 1(1 = x(2 x 1(2 = x(4 x 1(3 = x(6 x 2( = x(1 x 2(1 = x(3 x 2(2 = x(5 x 2(3 = x(7 N/2- point DFT N/2- point DFT X 1 ( X 1 (1 X 1 (2 X 1 (3 X 2 ( X 2 (1 X 2 (2 X 2 (3 W N 1 W N 2 W N 3 W N X ( X (1 X (2 X (3 X (4 X (5 X(6 X (7 N-point DFT CSE5311 Design and Analysis of Algorithms 24

25 x( x(4 x(2 x(6 x(1 x(5 x(3 x(7 W N W N W N W N W N 2 W N W N 2 W N W N 1 W N 2 W N 3 W N X ( X (1 X (2 X (3 X (4 X (5 X (6 X (7 CSE5311 Design and Analysis of Algorithms 25

26 Recursion Tree ( a, a1, a2, a3, a4, a5, a6, a7 ( a, a2, a4, a6 ( a1, a3, a5, a7 ( a, a4 ( a2, a6 ( a1, a5 ( a3, a7 ( a ( a4 ( a2 ( a6 ( a1 ( a5 ( a3 ( a7 1 We take the elements in pairs, compute the DFT of each pair, using one butterfly operation, and replace the pair with its DFT 2 We take these n/2 DFT s in pairs and compute the DFT of the four vector elements We take 2 (n/2-element DFT s and combine them using n/2 butterfly operations into the final n-element DFT CSE5311 Design and Analysis of Algorithms 26

27 Why Bit-reversed Order x( n n 2 1n n 1 n1 n x( x(1 x(1 x(11 x(1 x(11 x(11 x(111 x( x(4 x(2 x(6 x(1 x(5 x(3 x(7 CSE5311 Design and Analysis of Algorithms 27

28 Point-Value to Coefficient Representation: Inverse DFT Goal. Given the values y,..., y n-1 of a degree n-1 polynomial at the n points ω, ω 1,, ω n-1, find unique polynomial a + a 1 x a n-1 x n-1 that has given values at given points. CSE5311 Design and Analysis of Algorithms 28

29 Inverse DFT Inverse of Fourier matrix is given by following formula To compute inverse FFT, apply same algorithm but use ω -1 = e -2π i / n as principal n th root of unity (and divide by n. CSE5311 Design and Analysis of Algorithms 29

30 Inverse FFT CSE5311 Design and Analysis of Algorithms 3

31 Inverse FFT Theorem. Inverse FFT algorithm interpolates a degree n-1 polynomial given values at each of the n th roots of unity in O(n log n steps. CSE5311 Design and Analysis of Algorithms 31

32 Polynomial Multiplication Theorem: We can multiply two degree n-1 polynomials in O(n log n steps. CSE5311 Design and Analysis of Algorithms 32

33 A Parallel FFT Circuit ω 2 ω 4 ω 2 1 ω 4 ω 8 ω 2 1 ω 8 ω 2 ω 4 1 ω 4 2 ω 8 3 ω 8 CSE5311 Design and Analysis of Algorithms 33