DIVIDE AND CONQUER II

Transcription

1 DIVIDE AND CONQUER II master theorem integer multiplication matrix multiplication convolution and FFT SECTION 5.6

2 Fourier analysis Fourier theorem. [Fourier, Dirichlet, Riemann] Any sufficiently smooth) periodic function can be expressed as the sum of a series of sinusoids. t yt) = 2 π N k=1 sin kt k N =

3 Euler's identity Euler's identity. e ix = cos x + i sin x. Sinusoids. Sum of sine and cosines = sum of complex exponentials. 37

4 Time domain vs. frequency domain Signal. [touch tone button 1] yt) = 1 2 sin2π 697 t) sin2π 1209 t) Time domain. sound pressure Frequency domain. 0.5 amplitude Reference: Cleve Moler, Numerical Computing with MATLAB 38

5 Time domain vs. frequency domain Signal. [recording, 8192 samples per second] Magnitude of discrete Fourier transform. Reference: Cleve Moler, Numerical Computing with MATLAB 39

6 Fast Fourier transform FFT. Fast way to convert between time-domain and frequency-domain. Alternate viewpoint. Fast way to multiply and evaluate polynomials. we take this approach If you speed up any nontrivial algorithm by a factor of a million or so the world will beat a path towards finding useful applications for it. Numerical Recipes 40

7 Fast Fourier transform: applications Applications. Optics, acoustics, quantum physics, telecommunications, radar, control systems, signal processing, speech recognition, data compression, image processing, seismology, mass spectrometry, Digital media. [DVD, JPEG, MP3, H.264] Medical diagnostics. [MRI, CT, PET scans, ultrasound] Numerical solutions to Poisson's equation. Integer and polynomial multiplication. Shor's quantum factoring algorithm. The FFT is one of the truly great computational developments of [the 20th] century. It has changed the face of science and engineering so much that it is not an exaggeration to say that life as we know it would be very different without the FFT. Charles van Loan 41

8 Fast Fourier transform: brief history Gauss 1805, 1866). Analyzed periodic motion of asteroid Ceres. Runge-König 1924). Laid theoretical groundwork. Danielson-Lanczos 1942). Efficient algorithm, x-ray crystallography. Cooley-Tukey 1965). Monitoring nuclear tests in Soviet Union and tracking submarines. Rediscovered and popularized FFT. paper published only after IBM lawyers decided not to set precedent of patenting numerical algorithms conventional wisdom: nobody could make money selling software!) Importance not fully realized until advent of digital computers. 42

9 Polynomials: coefficient representation Polynomial. [coefficient representation] Ax) = a 0 + a 1 x + a 2 x 2 ++ a n 1 x n 1 Bx) = b 0 + b 1 x + b 2 x 2 ++ b n 1 x n 1 Add. On) arithmetic operations. Ax)+ Bx) = a 0 +b 0 )+a 1 +b 1 )x ++ a n 1 +b n 1 )x n 1 Evaluate. On) using Horner's method. Ax) = a 0 +xa 1 + xa 2 ++ xa n 2 + xa n 1 )))) Multiply convolve). On 2 ) using brute force. Ax) Bx) = 2n 2 c i x i, where c i = a j b i j i =0 i j =0 43

10 A modest Ph.D. dissertation title DEMONSTRATIO NOVA THEOREMATIS OMNEM FVNCTIONEM ALGEBRAICAM RATIONALEM INTEGRAM VNIVS VARIABILIS IN FACTORES REALES PRIMI VEL SECUNDI GRADVS RESOLVI POSSE AVCTORE CAROLO FRIDERICO GAVSS HELMSTADII APVD C. G. FLECKEISEN Quaelibet aequatio algebraica determinata reduci potest ad formam x m +Ax m-1 +Bx m-2 + etc. +M=0, ita vt m sit numerus integer positiuus. Si partem primam huius aequationis per X denotamus, aequationique X=0 per plures valores inaequales ipsius x satisfieri supponimus, puta ponendo x=α, x=β, x=γ etc. functio X per productum e factoribus x-α, x-β, x-γ etc. diuisibilis erit. Vice versa, si productum e pluribus factoribus simplicibus x- α, x-β, x-γ etc. functionem X metitur: aequationi X=0 satisfiet, aequando ipsam x cuicunque quantitatum α, β, γ etc. Denique si X producto ex m factoribus talibus simplicibus aequalis est siue omnes diuersi sint, siue quidam ex ipsis identici): alii factores simplices praeter hos functionem X metiri non poterunt. Quamobrem aequatio m ti gradus plures quam m radices habere nequit; simul vero patet, aequationem m ti gradus pauciores radices habere posse, etsi X in m factores simplices resolubilis sit: "New proof of the theorem that every algebraic rational integral function in one variable can be resolved into real factors of the first or the second degree." 44

11 Polynomials: point-value representation Fundamental theorem of algebra. A degree n polynomial with complex coefficients has exactly n complex roots. Corollary. A degree n 1 polynomial Ax) is uniquely specified by its evaluation at n distinct values of x y y j = Ax j ) x j x 45

12 Polynomials: point-value representation Polynomial. [point-value representation] Ax) : x 0, y 0 ),, x n-1, y n 1 ) Bx) : x 0, z 0 ),, x n-1, z n 1 ) Add. On) arithmetic operations. Ax)+ Bx) : x 0, y 0 + z 0 ),, x n-1, y n 1 + z n 1 ) Multiply convolve). On), but need 2n 1 points. Ax) Bx) : x 0, y 0 z 0 ),, x 2n-1, y 2n 1 z 2n 1 ) Evaluate. On 2 ) using Lagrange's formula. x x j ) n 1 j k Ax) = y k k=0 x k x j ) j k 46

13 Converting between two representations Tradeoff. Fast evaluation or fast multiplication. We want both! representation multiply evaluate coefficient On 2 ) On) point-value On) On 2 ) Goal. Efficient conversion between two representations all ops fast. a 0, a 1,..., a n-1 x 0, y 0 ),, x n 1, y n 1 ) coefficient representation point-value representation 47

14 Converting between two representations: brute force Coefficient point-value. Given a polynomial a 0 + a 1 x a n 1 x n 1, evaluate it at n distinct points x 0,..., x n 1. # $ y 0 y 1 y 2 y n 1 & ' = # 2 1 x 0 x x 1 x x 2 x 2 $ 2 1 x n 1 x n 1 x 0 n 1 x 1 n 1 x 2 n 1 n 1 x n 1 & # ' $ a 0 a 1 a 2 a n 1 & ' Running time. On 2 ) for matrix-vector multiply or n Horner's). 48

15 Converting between two representations: brute force Point-value coefficient. Given n distinct points x 0,..., x n 1 and values y 0,..., y n 1, find unique polynomial a 0 + a 1 x a n 1 x n 1, that has given values at given points. # $ y 0 y 1 y 2 y n 1 & ' = # 2 1 x 0 x x 1 x x 2 x 2 $ 2 1 x n 1 x n 1 x 0 n 1 x 1 n 1 x 2 n 1 n 1 x n 1 & # ' $ a 0 a 1 a 2 a n 1 & ' Vandermonde matrix is invertible iff xi distinct Running time. On 3 ) for Gaussian elimination. or On ) via fast matrix multiplication 49

16 Divide-and-conquer Decimation in frequency. Break up polynomial into low and high powers. Ax) = a 0 + a 1 x + a 2 x 2 + a 3 x 3 + a 4 x 4 + a 5 x 5 + a 6 x 6 + a 7 x 7. A lowx) = a 0 + a 1 x + a 2 x 2 + a 3 x 3. A high x) = a 4 + a 5 x + a 6 x 2 + a 7 x 3. Ax) = A lowx) + x 4 A high x). Decimation in time. Break up polynomial into even and odd powers. Ax) = a 0 + a 1 x + a 2 x 2 + a 3 x 3 + a 4 x 4 + a 5 x 5 + a 6 x 6 + a 7 x 7. A evenx) = a 0 + a 2 x + a 4 x 2 + a 6 x 3. A odd x) = a 1 + a 3 x + a 5 x 2 + a 7 x 3. Ax) = A evenx 2 ) + x A odd x 2 ). 50

17 Coefficient to point-value representation: intuition Coefficient point-value. Given a polynomial a 0 + a 1 x a n 1 x n 1, evaluate it at n distinct points x 0,..., x n 1. we get to choose which ones! Divide. Break up polynomial into even and odd powers. Ax) = a 0 + a 1 x + a 2 x 2 + a 3 x 3 + a 4 x 4 + a 5 x 5 + a 6 x 6 + a 7 x 7. A evenx) = a 0 + a 2 x + a 4 x 2 + a 6 x 3. A odd x) = a 1 + a 3 x + a 5 x 2 + a 7 x 3. Ax) = A evenx 2 ) + x A odd x 2 ). A-x) = A evenx 2 ) x A odd x 2 ). Intuition. Choose two points to be ±1. A 1) = A even 1) + 1 A odd 1). A-1) = A even1) 1 A odd 1). Can evaluate polynomial of degree n at 2 points by evaluating two polynomials of degree ½n at 1 point. 51

18 Coefficient to point-value representation: intuition Coefficient point-value. Given a polynomial a 0 + a 1 x a n 1 x n 1, evaluate it at n distinct points x 0,..., x n 1. we get to choose which ones! Divide. Break up polynomial into even and odd powers. Ax) = a 0 + a 1 x + a 2 x 2 + a 3 x 3 + a 4 x 4 + a 5 x 5 + a 6 x 6 + a 7 x 7. A evenx) = a 0 + a 2 x + a 4 x 2 + a 6 x 3. A odd x) = a 1 + a 3 x + a 5 x 2 + a 7 x 3. Ax) = A evenx 2 ) + x A odd x 2 ). A-x) = A even x 2 ) x A odd x 2 ). Intuition. Choose four complex points to be ±1, ±i. A 1) = A even 1) + 1 A odd 1). A-1) = A even1) 1 A odd 1). A i ) = A even -1) + i A odd -1). A -i ) = A even -1) i A odd -1). Can evaluate polynomial of degree n at 4 points by evaluating two polynomials of degree ½n at 2 point. 52

19 Discrete Fourier transform Coefficient point-value. Given a polynomial a 0 + a 1 x a n 1 x n 1, evaluate it at n distinct points x 0,..., x n 1. Key idea. Choose x k = ω k where ω is principal n th root of unity. # $ y 0 y 1 y 2 y 3 y n 1 & ' = # & 1 ω 1 ω 2 ω 3 ω n 1 1 ω 2 ω 4 ω 6 ω 2n 1) 1 ω 3 ω 6 ω 9 ω 3n 1) $ 1 ω n 1 ω 2n 1) ω 3n 1) ω n 1)n 1) ' # $ a 0 a 1 a 2 a 3 a n 1 & ' DFT Fourier matrix Fn 53

20 Roots of unity Def. An n th root of unity is a complex number x such that x n = 1. Fact. The n th roots of unity are: ω 0, ω 1,, ω n 1 where ω = e 2π i / n. Pf. ω k ) n = e 2π i k / n ) n = e π i ) 2k = -1) 2k = 1. Fact. The ½n th roots of unity are: ν 0, ν 1,, ν n/2 1 where ν = ω 2 = e 4π i / n. ω 2 = ν 1 = i ω 3 ω 1 ω 4 = ν 2 = -1 n = 8 ω 0 = ν 0 = 1 ω 5 ω 7 ω 6 = ν 3 = -i 54

21 Fast Fourier transform Goal. Evaluate a degree n 1 polynomial Ax) = a a n 1 x n 1 at its n th roots of unity: ω 0, ω 1,, ω n 1. Divide. Break up polynomial into even and odd powers. A evenx) = a 0 + a 2 x + a 4 x a n-2 x n/2 1. A odd x) = a 1 + a 3 x + a 5 x a n-1 x n/2 1. Ax) = A evenx 2 ) + x A odd x 2 ). Conquer. Evaluate A even x) and A odd x) at the ½n th roots of unity: ν 0, ν 1,, ν n/2 1. ν k = ω k ) 2 Combine. Aω k ) = A even ν k ) + ω k A odd ν k ), 0 k < n/2 Aω k+ ½n ) = A even ν k ) ω k A odd ν k ), 0 k < n/2 ν k = ω k + ½n ) 2 ω k+ ½n = -ω k 55

22 FFT: implementation FFT n, a0, a1, a2,, an 1) IF n = 1) RETURN a0. e0, e1,, en/2 1) FFT n/2, a0, a2, a4,, an 2). d0, d1,, dn/2 1) FFT n/2, a1, a3, a4,, an 1). FOR k = 0 TO n/2 1. ω k e 2πik/n. yk ek + ω k dk. yk + n/2 ek ω k dk. RETURN y0, y1, y2,, yn 1). 56

23 FFT: summary Theorem. The FFT algorithm evaluates a degree n 1 polynomial at each of the n th roots of unity in On log n) steps and On) extra space. Pf. Tn) = 2Tn/2) + Θn) Tn) = Θn logn) assumes n is a power of 2 a 0, a 1,..., a n-1 On log n) x 0, y 0 ),, x n 1, y n 1 ) coefficient representation??? point-value representation 57

24 FFT: recursion tree a 0, a 1, a 2, a 3, a 4, a 5, a 6, a 7 inverse perfect shuffle a 0, a 2, a 4, a 6 a 1, a 3, a 5, a 7 a 0, a 4 a 2, a 6 a 1, a 5 a 3, a 7 a 0 a 4 a 2 a 6 a 1 a 5 a 3 a "bit-reversed" order 58

25 Inverse discrete Fourier transform Point-value coefficient. Given n distinct points x 0,..., x n 1 and values y 0,..., y n 1, find unique polynomial a 0 + a 1 x a n 1 x n 1, that has given values at given points. # $ a 0 a 1 a 2 a 3 a n 1 & ' = # & 1 ω 1 ω 2 ω 3 ω n 1 1 ω 2 ω 4 ω 6 ω 2n 1) 1 ω 3 ω 6 ω 9 ω 3n 1) $ 1 ω n 1 ω 2n 1) ω 3n 1) ω n 1)n 1) ' 1 # $ y 0 y 1 y 2 y 3 y n 1 & ' Inverse DFT Fourier matrix inverse Fn) -1 59

26 Inverse discrete Fourier transform Claim. Inverse of Fourier matrix F n is given by following formula: G n = 1 n $ ' & 1 ω 1 ω 2 ω 3 ω n 1) ) & ) & 1 ω 2 ω 4 ω 6 ω 2n 1) ) & 1 ω 3 ω 6 ω 9 ω 3n 1) ) & ) & ) & 1 ω n 1) ω 2n 1) ω 3n 1) ω n 1)n 1) ) Fn / n is a unitary matrix Consequence. To compute inverse FFT, apply same algorithm but use ω 1 = e 2π i / n as principal n th root of unity and divide the result by n). 60

27 Inverse FFT: proof of correctness Claim. F n and G n are inverses. Pf. F n G n ) k k " = 1 n n 1 ω k j ω j k " = 1 n j=0 n 1 ω k k ") j j=0 = & ' 1 if k = k " 0 otherwise summation lemma below) Summation lemma. Let ω be a principal n th root of unity. Then n 1 ω k j j=0 = & ' n if k 0 mod n 0 otherwise Pf. If k is a multiple of n then ω k = 1 series sums to n. Each n th root of unity ω k is a root of x n 1 = x 1) 1 + x + x x n-1 ). if ω k 1 we have: 1 + ω k + ω k2) + + ω kn-1) = 0 series sums to 0. 61

28 Inverse FFT: implementation Note. Need to divide result by n. INVERSE-FFT n, a0, a1, a2,, an 1) IF n = 1) RETURN a0. e0, e1,, en/2 1) INVERSE-FFT n/2, a0, a2, a4,, an 2). d0, d1,, dn/2 1) INVERSE-FFT n/2, a1, a3, a4,, an 1). FOR k = 0 TO n/2 1. ω k e 2πik/n. yk ek + ω k dk ). yk + n/2 ek ω k dk ). RETURN y0, y1, y2,, yn 1). 62

29 Inverse FFT: summary Theorem. The inverse FFT algorithm interpolates a degree n 1 polynomial given values at each of the n th roots of unity in On log n) steps. assumes n is a power of 2 On log n) FFT) a 0, a 1,..., a n-1 coefficient representation On log n) inverse FFT) x 0, y 0 ),, x n 1, y n 1 ) point-value representation 63

30 Polynomial multiplication Theorem. Can multiply two degree n 1 polynomials in On log n) steps. Pf. pad with 0s to make n a power of 2 coefficient representation coefficient representation a 0, a 1,, a n-1 b 0, b 1,, b n-1 c 0, c 1,, c 2n-2 2 FFTs On log n) inverse FFT On log n) Aω 0 ),..., Aω 2n 1 ) Bω 0 ),..., Bω 2n 1 ) point-value multiplication On) Cω 0 ),..., Cω 2n 1 ) 64

31 FFT in practice? 65

32 FFT in practice Fastest Fourier transform in the West. [Frigo and Johnson] Optimized C library. Features: DFT, DCT, real, complex, any size, any dimension. Won 1999 Wilkinson Prize for Numerical Software. Portable, competitive with vendor-tuned code. Implementation details. Core algorithm is nonrecursive version of Cooley-Tukey. Instead of executing predetermined algorithm, it evaluates your hardware and uses a special-purpose compiler to generate an optimized algorithm catered to "shape" of the problem. Runs in On log n) time, even when n is prime. Multidimensional FFTs. 66

33 Integer multiplication, redux Integer multiplication. Given two n-bit integers a = a n 1 a 1 a 0 and b = b n 1 b 1 b 0, compute their product a b. Convolution algorithm. Form two polynomials. Note: a = A2), b = B2). Compute Cx) = Ax) Bx). Evaluate C2) = a b. Ax) = a 0 + a 1 x + a 2 x 2 ++ a n 1 x n 1 Bx) = b 0 +b 1 x +b 2 x 2 ++ b n 1 x n 1 Running time: On log n) complex arithmetic operations. Theory. [Schönhage-Strassen 1971] On log n log log n) bit operations. Theory. [Fürer 2007] n log n 2 Olog* n) bit operations. Practice. [GNU Multiple Precision Arithmetic Library] It uses brute force, Karatsuba, and FFT, depending on the size of n. 67