5.6 Convolution and FFT

Transcription

1 5.6 Convolution and FFT

2 Fast Fourier Transform: Applications Applications. Optics, acoustics, quantum physics, telecommunications, control systems, signal processing, speech recognition, data compression, image processing. DVD, JPEG, MP3, MRI, CAT scan. Numerical solutions to Poisson's equation. The FFT is one of the truly great computational developments of this [20th] century. It has changed the face of science and engineering so much that it is not an exaggeration to say that life as we know it would be very different without the FFT. -Charles van Loan 3

3 Fast Fourier Transform: Brief History Gauss (1805, 1866). Analyzed periodic motion of asteroid Ceres. Runge-König (1924). Laid theoretical groundwork. Danielson-Lanczos (1942). Efficient algorithm. Cooley-Tukey (1965). Monitoring nuclear tests in Soviet Union and tracking submarines. Rediscovered and popularized FFT. Importance not fully realized until advent of digital computers. 4

4 Polynomials: Coefficient Representation Polynomial. [coefficient representation] A(x) = a 0 + a 1 x + a 2 x 2 +L+ a n 1 x n 1 B(x) = b 0 + b 1 x + b 2 x 2 +L+ b n 1 x n 1 Add: O(n) arithmetic operations. A(x)+ B(x) = (a 0 +b 0 )+(a 1 +b 1 )x +L+ (a n 1 +b n 1 )x n 1 Evaluate: O(n) using Horner's method. A(x) = a 0 +(x(a 1 + x(a 2 +L+ x(a n 2 + x(a n 1 ))L)) Multiply (convolve): O(n 2 ) using brute force. A(x) B(x) = 2n 2 c i x i, where c i = a j b i j i =0 i j =0 5

5 Polynomials: Point-Value Representation Fundamental theorem of algebra. [Gauss, PhD thesis] A degree n polynomial with complex coefficients has n complex roots. Corollary. A degree n-1 polynomial A(x) is uniquely specified by its evaluation at n distinct values of x. y y j = A(x j ) x j x 6

6 Polynomials: Point-Value Representation Polynomial. [point-value representation] A(x) : (x 0, y 0 ), K, (x n-1, y n 1 ) B(x) : (x 0, z 0 ), K, (x n-1, z n 1 ) Add: O(n) arithmetic operations. A(x)+ B(x) : (x 0, y 0 + z 0 ),K, (x n-1, y n 1 + z n 1 ) Multiply: O(n), but need 2n-1 points. A(x) B(x) : (x 0, y 0 z 0 ),K, (x 2n-1, y 2n 1 z 2n 1 ) Evaluate: O(n 2 ) using Lagrange's formula. (x x j ) n 1 j k A(x) = y k (x k x j ) k=0 j k 7

7 Converting Between Two Polynomial Representations Tradeoff. Fast evaluation or fast multiplication. We want both! Representation Coefficient Multiply O(n 2 ) Evaluate O(n) Point-value O(n) O(n 2 ) Goal. Make all ops fast by efficiently converting between two representations. a 0, a 1,K, a n-1 (x 0, y 0 ), K, (x n 1, y n 1 ) coefficient representation point-value representation 8

8 Converting Between Two Polynomial Representations: Brute Force Coefficient to point-value. Given a polynomial a 0 + a 1 x a n-1 x n-1, evaluate it at n distinct points x 0,..., x n-1. y 0 y 1 y 2 M y n 1 = 2 1 x 0 x 0 L 2 1 x 1 x 1 L 2 1 x 2 x 2 L M M M O M L 2 1 x n 1 x n 1 x 0 n 1 x 1 n 1 x 2 n 1 n 1 x n 1 a 0 a 1 a 2 M a n 1 O(n 2 ) for matrix-vector multiply O(n 3 ) for Gaussian elimination Vandermonde matrix is invertible iff x i distinct Point-value to coefficient. Given n distinct points x 0,..., x n-1 and values y 0,..., y n-1, find unique polynomial a 0 + a 1 x a n-1 x n-1 that has given values at given points. 9

9 Coefficient to Point-Value Representation: Intuition Coefficient to point-value. Given a polynomial a 0 + a 1 x a n-1 x n-1, evaluate it at n distinct points x 0,..., x n-1. Divide. Break polynomial up into even and odd powers. A(x) = a 0 + a 1 x + a 2 x 2 + a 3 x 3 + a 4 x 4 + a 5 x 5 + a 6 x 6 + a 7 x 7. A even (x) = a 0 + a 2 x + a 4 x 2 + a 6 x 3. A odd (x) = a 1 + a 3 x + a 5 x 2 + a 7 x 3. A(-x) = A even (x 2 ) + x A odd (x 2 ). A(-x) = A even (x 2 ) - x A odd (x 2 ). Intuition. Choose two points to be ±1. A(-1) = A even (1) + 1 A odd (1). A(-1) = A even (1) - 1 A odd (1). Can evaluate polynomial of degree n at 2 points by evaluating two polynomials of degree ½n at 1 point. 10

10 Coefficient to Point-Value Representation: Intuition Coefficient to point-value. Given a polynomial a 0 + a 1 x a n-1 x n-1, evaluate it at n distinct points x 0,..., x n-1. Divide. Break polynomial up into even and odd powers. A(x) = a 0 + a 1 x + a 2 x 2 + a 3 x 3 + a 4 x 4 + a 5 x 5 + a 6 x 6 + a 7 x 7. A even (x) = a 0 + a 2 x + a 4 x 2 + a 6 x 3. A odd (x) = a 1 + a 3 x + a 5 x 2 + a 7 x 3. A(-x) = A even (x 2 ) + x A odd (x 2 ). A(-x) = A even (x 2 ) - x A odd (x 2 ). Intuition. Choose four points to be ±1, ±i. A(-1) = A even (-1) + 1 A odd ( 1). A(-1) = A even (-1) - 1 A odd (-1). A(-i) = A even (-1) + i A odd (-1). A(-i) = A even (-1) - i A odd (-1). Can evaluate polynomial of degree n at 4 points by evaluating two polynomials of degree ½n at 2 points. 11

11 Discrete Fourier Transform Coefficient to point-value. Given a polynomial a 0 + a 1 x a n-1 x n-1, evaluate it at n distinct points x 0,..., x n-1. Key idea: choose x k = ω k where ω is principal n th root of unity. y 0 y 1 y 2 y 3 M y n 1 = L 1 1 ω 1 ω 2 ω 3 L ω n 1 1 ω 2 ω 4 ω 6 L ω 2(n 1) 1 ω 3 ω 6 ω 9 L ω 3(n 1) M M M M O M 1 ω n 1 ω 2(n 1) ω 3(n 1) L ω (n 1)(n 1) a 0 a 1 a 2 a 3 M a n 1 Discrete Fourier transform Fourier matrix F n 12

12 Roots of Unity Def. An n th root of unity is a complex number x such that x n = 1. Fact. The n th roots of unity are: ω 0, ω 1,, ω n-1 where ω = e 2π i / n. Pf. (ω k ) n = (e 2π i k / n ) n = (e π i ) 2k = (-1) 2k = 1. Fact. The ½n th roots of unity are: ν 0, ν 1,, ν n/2-1 where ν = e 4π i / n. Fact. ω 2 = ν and (ω 2 ) k = ν k. ω 2 = ν 1 = i ω 3 ω 1 ω 4 = ν 2 = -1 n = 8 ω 0 = ν 0 = 1 ω 5 ω 7 ω 6 = ν 3 = -i 13

13 Fast Fourier Transform Goal. Evaluate a degree n-1 polynomial A(x) = a a n-1 x n-1 at its n th roots of unity: ω 0, ω 1,, ω n-1. Divide. Break polynomial up into even and odd powers. A even (x) = a 0 + a 2 x + a 4 x a n/2-2 x (n-1)/2. A odd (x) = a 1 + a 3 x + a 5 x a n/2-1 x (n-1)/2. A(x) = A even (x 2 ) + x A odd (x 2 ). Conquer. Evaluate degree A even (x) and A odd (x) at the ½n th roots of unity: ν 0, ν 1,, ν n/2-1. Combine. A(ω k+n ) = A even (ν k ) + ω k A odd (ν k ), 0 k < n/2 A(ω k+n ) = A even (ν k ) - ω k A odd (ν k ), 0 k < n/2 ν k = (ω k ) 2 = (ω k+n ) 2 ω k+n = -ω k 14

14 FFT Algorithm fft(n, a 0,a 1,,a n-1 ) { if (n == 1) return a 0 (e 0,e 1,,e n/2-1 ) FFT(n/2, a 0,a 2,a 4,,a n-2 ) (d 0,d 1,,d n/2-1 ) FFT(n/2, a 1,a 3,a 5,,a n-1 ) for k = 0 to n/2-1 { ω k e 2πik/n y k+n/2 e k + ω k d k } y k+n/2 e k - ω k d k } return (y 0,y 1,,y n-1 ) 15

15 FFT Summary Theorem. FFT algorithm evaluates a degree n-1 polynomial at each of the n th roots of unity in O(n log n) steps. assumes n is a power of 2 Running time. T(2n) = 2T(n) + O(n) T(n) = O(n log n). O(n log n) a 0, a 1,K, a n-1 (ω 0, y 0 ), K, (ω n 1, y n 1 ) coefficient representation point-value representation 16

16 Recursion Tree a 0, a 1, a 2, a 3, a 4, a 5, a 6, a 7 perfect shuffle a 0, a 2, a 4, a 6 a 1, a 3, a 5, a 7 a 0, a 4 a 2, a 6 a 1, a 5 a 3, a 7 a 0 a 4 a 2 a 6 a 1 a 5 a 3 a "bit-reversed" order 17

17 Point-Value to Coefficient Representation: Inverse DFT Goal. Given the values y 0,..., y n-1 of a degree n-1 polynomial at the n points ω 0, ω 1,, ω n-1, find unique polynomial a 0 + a 1 x a n-1 x n-1 that has given values at given points. a 0 a 1 a 2 a 3 M a n 1 = L 1 1 ω 1 ω 2 ω 3 L ω n 1 1 ω 2 ω 4 ω 6 L ω 2(n 1) 1 ω 3 ω 6 ω 9 L ω 3(n 1) M M M M O M 1 ω n 1 ω 2(n 1) ω 3(n 1) L ω (n 1)(n 1) y 0 y 1 y 2 y 3 M y n 1 Inverse DFT Fourier matrix inverse (F n ) -1 18

18 Inverse FFT Claim. Inverse of Fourier matrix is given by following formula. G n = 1 n L 1 1 ω 1 ω 2 ω 3 L ω (n 1) 1 ω 2 ω 4 ω 6 L ω 2(n 1) 1 ω 3 ω 6 ω 9 L ω 3(n 1) M M M M O M 1 ω (n 1) ω 2(n 1) ω 3(n 1) L ω (n 1)(n 1) Consequence. To compute inverse FFT, apply same algorithm but use ω -1 = e -2π i / n as principal n th root of unity (and divide by n). 19

19 Inverse FFT: Proof of Correctness Claim. F n and G n are inverses. Pf. ( F n G n ) k k = 1 n n 1 ω k j ω j k = 1 n j=0 n 1 ω (k k ) j j=0 = 1 if k = k 0 otherwise summation lemma Summation lemma. Let ω be a principal n th root of unity. Then n 1 ω k j j=0 = n if k 0 mod n 0 otherwise Pf. If k is a multiple of n then ω k = 1 sums to n. Each n th root of unity ω k is a root of x n - 1 = (x - 1) (1 + x + x x n-1 ). if ω k 1 we have: 1 + ω k + ω k(2) ω k(n-1) = 0 sums to 0. 20

20 Inverse FFT: Algorithm ifft(n, a 0,a 1,,a n-1 ) { if (n == 1) return a 0 (e 0,e 1,,e n/2-1 ) FFT(n/2, a 0,a 2,a 4,,a n-2 ) (d 0,d 1,,d n/2-1 ) FFT(n/2, a 1,a 3,a 5,,a n-1 ) for k = 0 to n/2-1 { } ω k e -2πik/n y k+n/2 (e k + ω k d k ) / n y k+n/2 (e k - ω k d k ) / n } return (y 0,y 1,,y n-1 ) 21

21 Inverse FFT Summary Theorem. Inverse FFT algorithm interpolates a degree n-1 polynomial given values at each of the n th roots of unity in O(n log n) steps. assumes n is a power of 2 O(n log n) a 0, a 1,K, a n-1 (ω 0, y 0 ), K, (ω n 1, y n 1 ) coefficient representation O(n log n) point-value representation 22

22 Polynomial Multiplication Theorem. Can multiply two degree n-1 polynomials in O(n log n) steps. coefficient representation coefficient representation a 0, a 1,K, a n-1 b 0, b 1,K, b n-1 c 0, c 1,K, c 2n-2 FFT O(n log n) inverse FFT O(n log n) A(x 0 ),K, A(x 2n-1 ) point-value multiplication B(x 0 ),K, B(x 2n-1 ) C(x 0 ), C(x 1 ),K, C(x 2n-1 ) O(n) 23

23 FFT in Practice Fastest Fourier transform in the West. [Frigo and Johnson] Optimized C library. Features: DFT, DCT, real, complex, any size, any dimension. Won 1999 Wilkinson Prize for Numerical Software. Portable, competitive with vendor-tuned code. Implementation details. Instead of executing predetermined algorithm, it evaluates your hardware and uses a special-purpose compiler to generate an optimized algorithm catered to "shape" of the problem. Core algorithm is nonrecursive version of Cooley-Tukey radix 2 FFT. O(n log n), even for prime sizes. Reference: 24