How to Multiply. 5.5 Integer Multiplication. Complex Multiplication. Integer Arithmetic. Complex multiplication. (a + bi) (c + di) = x + yi.

Transcription

1 How to ultiply Slides by Kevin Wayne. Copyright 5 Pearson-Addison Wesley. All rights reserved. integers, matrices, and polynomials Complex ultiplication Complex multiplication. a + bi) c + di) = x + yi. Grade-school. x = a!c - b!d, y = b!c + a!d. Gauss. x = a!c - b!d, y = a + b)! c + d) - a!c - b!d. Remark. Improvement if no hardware multiply. 4 multiplications, additions 3 multiplications, 5 additions 5.5 Integer ultiplication 4 Integer Arithmetic Add. Given two n-bit integers a and b, compute a + b. Grade-school. "n) bit operations. ultiply. Given two n-bit integers a and b, compute a! b. Grade-school. "n ) bit operations. * + Add ultiply

2 Divide-and-Conquer ultiplication: Warmup Karatsuba ultiplication To multiply two n-bit integers:! ultiply four "n-bit integers.! Add two "n-bit integers, and shift to obtain result. To multiply two n-bit integers:! Add two "n bit integers.! ultiply three "n-bit integers.! Add, subtract, and shift "n-bit integers to obtain result. Ex. x = n / " x + x y = n / " y + y ) n / " y + y ) = n " x y + n / " x y + x y xy = n / " x + x x = y = x x y y ) Tn) = 4T n/ "n) 3 # Tn) = "n ) recursive calls add, shift ) + x y x = n / " x + x y = n / " y + y xy = n " x y + n / " x y + x y ) + x y ) + x y = n " x y + n / " x + x )y + y ) # x y # x y Theorem. [Karatsuba-Ofman, 96] Can multiply two n-bit integers in On.585 ) bit operations. ) + T n/ ) + T + n/ ) Tn) " T # n/ + n) Tn) = On log 3 ) = On ) recursive calls 3 3 add, subtract, shift 5 6 Karatsuba: Recursion Tree Fast Integer Division Too!) " if n = Tn) = # 3Tn/) + n otherwise log n Tn) = " n 3 = k= ) k ) +log n # 3 3 # = 3n log 3 # Integer division. Given two n-bit or less) integers a and b, compute quotient q = a / b and remainder r = a mod b. Tn/) Tn) Tn/) Tn/) n 3n/) Fact. Complexity of integer division is same as multiplication. To compute quotient q: x i+ = x i " b x! Approximate x = / b using Newtons method: i! After log n iterations, either q = #a x or q = a x. using fast multiplication Tn/4) Tn/4) Tn/4) Tn/4) Tn/4) Tn/4) Tn/4) Tn/4) Tn/4) 9n/4) Tn / k ) 3 k n / k ) T) T) T) T) T) T) T) T) 3 lg n ) 7 8

3 atrix ultiplication atrix ultiplication atrix multiplication. Given two n-by-n matrices A and B, compute C = AB. " c c L c n c c L c n O # c n c n L c nn = " a a L a n a a L a n O # a n a n L a nn " b b L b n b b L b n O # b n b n L b nn n c ij = " a ik b kj k= j Ex. " # i ".7.. =.3.6. #.5..4 " # Brute force. "n 3 ) arithmetic operations. Fundamental question. Can we improve upon brute force? atrix ultiplication: Warmup atrix ultiplication: Key Idea Divide-and-conquer.! Divide: partition A and B into "n-by-"n blocks.! Conquer: multiply 8 "n-by-"n recursively.! Combine: add appropriate products using 4 matrix additions. Key idea. multiply -by- block matrices with only 7 multiplications. " C C " = A A " B B P = A " B # B ) # C C # A A # B B P = A + A ) " B " C C " = A A " B B # C C # A A # B B ) + A " B ) ) + A " B ) ) + A " B ) ) + A " B ) C = A " B C = A " B C = A " B C = A " B C = P 5 + P 4 " P + P 6 C = P + P C = P 3 + P 4 C = P 5 + P " P 3 " P 7 P 3 = A + A ) " B P 4 = A " B # B ) P 5 = A + A ) " B + B ) P 6 = A # A ) " B + B ) P 7 = A # A ) " B + B ) ) Tn) = 8T n/ "n ) recursive calls add, form submatrices # Tn) = "n 3 )! 7 multiplications.! 8 = 8 + additions and subtractions).

4 Fast atrix ultiplication Fast atrix ultiplication in Practice Fast matrix multiplication. [Strassen, 969]! Divide: partition A and B into "n-by-"n blocks.! Compute: 4 "n-by-"n matrices via matrix additions.! Conquer: multiply 7 pairs of "n-by-"n matrices recursively.! Combine: 7 products into 4 terms using 8 matrix additions. Analysis.! Assume n is a power of.! Tn) = # arithmetic operations. ) Tn) = 7T n / "n ) # Tn) = "n 4 43 log 7 ) = On.8 ) recursive calls add, subtract Implementation issues.! Sparsity.! Caching effects.! Numerical stability.! Odd matrix dimensions.! Crossover to classical algorithm around n = 8. Common misperception: "Strassen is only a theoretical curiosity."! Advanced Computation Group at Apple Computer reports 8x speedup on G4 Velocity Engine when n,5.! Range of instances where its useful is a subject of controversy. Remark. Can "Strassenize" Ax=b, determinant, eigenvalues,. 3 4 Fast atrix ultiplication in Theory Fast atrix ultiplication in Theory Q. ultiply two -by- matrices with 7 scalar multiplications? A. Yes! [Strassen, 969] "n log 7 ) = On.87 ) Q. ultiply two -by- matrices with 6 scalar multiplications? A. Impossible. [Hopcroft and Kerr, 97] "n log 6 ) = On.59 ) Q. Two 3-by-3 matrices with scalar multiplications? A. Also impossible. "n log 3 ) = On.77 ) Best known. On.376 ) [Coppersmith-Winograd, 987] Conjecture. On + ) for any >. Caveat. Theoretical improvements to Strassen are progressively less practical. Begun, the decimal wars have. [Pan, Bini et al, Schönhage, ]! Two -by- matrices with 4,46 scalar multiplications. On.85 )! Two 48-by-48 matrices with 47,7 scalar multiplications. On.78 )! A year later. On.7799 )! December, 979. On.583 )! January, 98. On.58 ) 5 6

5 Time Domain vs. Frequency Domain 5.6 Convolution and FFT Signal. [touch tone button ] yt) = sin" # 697 t) + sin" # 9 t) Time domain. sound pressure time seconds) Frequency domain..5 amplitude frequency Hz) Reference: Cleve oler, Numerical Computing with ATLAB 8 Time Domain vs. Frequency Domain Fast Fourier Transform Signal. [recording, 89 samples per second] FFT. Fast way to convert between time-domain and frequency-domain. Alternate viewpoint. Fast way to add, multiply, and evaluate polynomials. we take this approach agnitude of discrete Fourier transform. If you speed up any nontrivial algorithm by a factor of a million or so the world will beat a path towards finding useful applications for it. -Numerical Recipes Reference: Cleve oler, Numerical Computing with ATLAB 9

6 Fast Fourier Transform: Applications Fast Fourier Transform: Brief History Applications.! Optics, acoustics, quantum physics, telecommunications, radar, control systems, signal processing, speech recognition, data compression, image processing, seismology, mass spectrometry! Digital media. [DVD, JPEG, P3, H.64]! edical diagnostics. [RI, CT, PET scans, ultrasound]! Numerical solutions to Poissons equation.! Shors quantum factoring algorithm.! Gauss 85, 866). Analyzed periodic motion of asteroid Ceres. Runge-König 94). Laid theoretical groundwork. Danielson-Lanczos 94). Efficient algorithm, x-ray crystallography. Cooley-Tukey 965). onitoring nuclear tests in Soviet Union and tracking submarines. Rediscovered and popularized FFT. The FFT is one of the truly great computational developments of [the th] century. It has changed the face of science and engineering so much that it is not an exaggeration to say that life as we know it would be very different without the FFT. -Charles van Loan Importance not fully realized until advent of digital computers. Polynomials: Coefficient Representation Polynomials: Point-Value Representation Polynomial. [coefficient ] Ax) = a + a x + a x +L+ a n" x n" Bx) = b + b x + b x +L+ b n" x n" Add. On) arithmetic operations. Ax)+ Bx) = a +b )+a +b )x +L+ a n" +b n" )x n" Fundamental theorem of algebra. [Gauss, PhD thesis] A degree n polynomial with complex coefficients has exactly n complex roots. Corollary. A degree n- polynomial Ax) is uniquely specified by its evaluation at n distinct values of x. y Evaluate. On) using Horners method. Ax) = a +xa + xa +L+ xa n" + xa n" ))L)) ultiply convolve). On ) using brute force. n# Ax)" Bx) = c i x i, where c i = a j b i# j i = j = i y j = Ax j ) x j x 3 5

7 Polynomials: Point-Value Representation Converting Between Two Polynomial Representations Polynomial. [point-value ] Ax) : x, y ), K, x n-, y n" ) Bx) : x, z ), K, x n-, z n" ) Add. On) arithmetic operations. Ax)+ Bx) : x, y + z ),K, x n-, y n" + z n" ) Tradeoff. Fast evaluation or fast multiplication. We want both! Representation ultiply Evaluate Coefficient On ) On) Point-value On) On ) ultiply convolve). On), but need n- points. Ax) " Bx) : x, y " z ),K, x n-, y n# " z n# ) Goal. ake all ops fast by efficiently converting between two s. Evaluate. On ) using Lagranges formula. x " x j ) n" j#k Ax) = y k k= x k " x j ) j#k a, a,k, a n- x, y ), K, x n", y n" ) coefficient point-value 6 7 Converting Between Two Representations: Brute Force Coefficient to Point-Value Representation: Intuition Coefficient ) point-value. Given a polynomial a + a x a n- x n-, evaluate it at n distinct points x,..., x n-. Coefficient ) point-value. Given a polynomial a + a x a n- x n-, evaluate it at n distinct points x,..., x n-. we get to choose which ones! # y y y y n" = # n" x x L x # a n" x x L x a n" x x L x a O n" x n" x n" L x n" a n" On ) via matrix-vector multiply On 3 ) via Gaussian elimination Divide. Break polynomial up into even and odd powers.! Ax) = a + a x + a x + a 3 x 3 + a 4 x 4 + a 5 x 5 + a 6 x 6 + a 7 x 7.! A even x) = a + a x + a 4 x + a 6 x 3.! A odd x) = a + a 3 x + a 5 x + a 7 x 3.! A-x) = A even x ) + x A odd x ).! A-x) = A even x ) - x A odd x ). Vandermonde matrix is invertible iff x i distinct Point-value ) coefficient. Given n distinct points x,..., x n- and values y,..., y n-, find unique polynomial a + a x a n- x n- that has given values at given points. Intuition. Choose two points to be ±.! A-) = A even ) + A odd ).! A-) = A even ) - A odd ). Can evaluate polynomial of degree * n at points by evaluating two polynomials of degree * "n at point. 8 9

8 Coefficient to Point-Value Representation: Intuition Discrete Fourier Transform Coefficient ) point-value. Given a polynomial a + a x a n- x n-, evaluate it at n distinct points x,..., x n-. Divide. Break polynomial up into even and odd powers.! Ax) = a + a x + a x + a 3 x 3 + a 4 x 4 + a 5 x 5 + a 6 x 6 + a 7 x 7.! A even x) = a + a x + a 4 x + a 6 x 3.! A odd x) = a + a 3 x + a 5 x + a 7 x 3.! A-x) = A even x ) + x A odd x ).! A-x) = A even x ) - x A odd x ). Intuition. Choose four points to be ±, ±i.! A-) = A even -) + A odd ).! A-) = A even -) - A odd -).! A-i) = A even -) + i A odd -).! A-i) = A even -) - i A odd -). we get to choose which ones! Can evaluate polynomial of degree * n at 4 points by evaluating two polynomials of degree * "n at points. Coefficient ) point-value. Given a polynomial a + a x a n- x n-, evaluate it at n distinct points x,..., x n-. Key idea. Choose x k = + k where + is principal n th root of unity. # y y y y 3 y n" = Discrete Fourier transform # L # ) ) ) 3 L ) n" ) ) 4 ) 6 L ) n") ) 3 ) 6 ) 9 L ) 3n") O ) n" ) n") ) 3n") L ) n")n") Fourier matrix F n a a a a 3 a n" 3 3 Roots of Unity Fast Fourier Transform Def. An n th root of unity is a complex number x such that x n =. Fact. The n th roots of unity are: +, +,, + n- where + = e, i / n. Pf. + k ) n = e, i k / n ) n = e, i ) k = -) k =. Fact. The "n th roots of unity are: -, -,, - n/- where - = e 4, i / n. Note. + = - and + ) k = - k = - = i + Goal. Evaluate a degree n- polynomial Ax) = a a n- x n- at its n th roots of unity: +, +,, + n-. Divide. Break up polynomial into even and odd powers.! A even x) = a + a x + a 4 x + + a n/- x n-)/.! A odd x) = a + a 3 x + a 5 x + + a n/- x n-)/.! Ax) = A even x ) + x A odd x ). Conquer. Evaluate A even x) and A odd x) at the "n th roots of unity: -, -,, - n/ = - = - n = 8 + = - = Combine.! A+ k+n ) = A even - k ) + + k A odd - k ), * k < "n! A+ k+ "n ) = A even - k ) - + k A odd - k ), * k < "n - k = + k ) = + k+ "n ) = - 3 = -i k+ "n = -+ k 3 33

9 FFT Algorithm FFT Summary fftn, a,a,,a n- ) { if n == ) return a e,e,,e n/- ). FFTn/, a,a,a 4,,a n- ) d,d,,d n/- ). FFTn/, a,a 3,a 5,,a n- ) Theorem. FFT algorithm evaluates a degree n- polynomial at each of the n th roots of unity in On log n) steps. assumes n is a power of Running time. Tn) = Tn /) + "n) # Tn) = "n log n) for k = to n/ - { + k. e,ik/n } y k+n/. e k + + k d k y k+n/. e k - + k d k On log n) } return y,y,,y n- ) a, a,k, a n- ", y ), K, " n#, y n# ) coefficient point-value Recursion Tree Point-Value to Coefficient Representation: Inverse DFT a, a, a, a 3, a 4, a 5, a 6, a 7 Goal. Given the values y,..., y n- of a degree n- polynomial at the n points +, +,, + n-, find unique) polynomial a + a x a n- x n- that has given values at given points. perfect shuffle a, a, a 4, a 6 a, a 4 a, a 6 a, a 3, a 5, a 7 a, a 5 a 3, a 7 # a a a a 3 a n" = # L " ) ) ) 3 L ) n" ) ) 4 ) 6 L ) n") ) 3 ) 6 ) 9 L ) 3n") O ) n" ) n") ) 3n") L ) n")n") # y y y y 3 y n" a a 4 a a 6 a a 5 a 3 a 7 Inverse DFT Fourier matrix inverse F n ) - "bit-reversed" order 36 37

10 Inverse FFT Inverse FFT: Proof of Correctness Claim. Inverse of Fourier matrix is given by following formula. G n L " # " # " #3 L " #n#) ) ) = " # " #4 " #6 L " #n#) ) n " #3 " #6 " #9 L " #3n#) ) ) O ) " #n#) " #n#) " #3n#) L " #n#)n#) ) Consequence. To compute inverse FFT, apply same algorithm but use + - = e -, i / n as principal n th root of unity and divide by n). Claim. F n and G n are inverses. Pf. Summation lemma. Let + be a principal n th root of unity. Then Pf. F n G n ) k k " = n n# " k j j= = n if k mod n otherwise! If k is a multiple of n then + k = ) sums to n. n j= n # k k ") j j= # k j # j k " = = n! Each n th root of unity + k is a root of x n - = x - ) + x + x x n- ).! if + k / we have: + + k + + k) kn-) = ) sums to.! if k = k " otherwise summation lemma Inverse FFT: Algorithm Inverse FFT Summary ifftn, a,a,,a n- ) { if n == ) return a Theorem. Inverse FFT algorithm interpolates a degree n- polynomial given values at each of the n th roots of unity in On log n) steps. assumes n is a power of e,e,,e n/- ). FFTn/, a,a,a 4,,a n- ) d,d,,d n/- ). FFTn/, a,a 3,a 5,,a n- ) } for k = to n/ - { + k. e -,ik/n } y k+n/. e k + + k d k ) / n y k+n/. e k - + k d k ) / n return y,y,,y n- ) a, a,k, a n- ", y ), K, " n#, y n# ) coefficient On log n) On log n) point-value 4 4

11 Polynomial ultiplication FFT in Practice Theorem. Can multiply two degree n- polynomials in On log n) steps. coefficient coefficient a, a,k, a n- b, b,k, b n- c, c,k, c n- FFT On log n) inverse FFT On log n) Fastest Fourier transform in the West. [Frigo and Johnson]! Optimized C library.! Features: DFT, DCT, real, complex, any size, any dimension.! Won 999 Wilkinson Prize for Numerical Software.! Portable, competitive with vendor-tuned code. Implementation details.! Instead of executing predetermined algorithm, it evaluates your hardware and uses a special-purpose compiler to generate an optimized algorithm catered to "shape" of the problem.! Core algorithm is nonrecursive version of Cooley-Tukey radix FFT.! On log n), even for prime sizes. Ax ),K, Ax n- ) point-value multiplication Bx ),K, Bx n- ) Cx ), Cx ),K, Cx n- ) On) Reference: Integer ultiplication, Redux Integer multiplication. Given two n bit integers a = a n- a a and b = b n- b b, compute their product a! b. Convolution algorithm. Ax) = a + a x + a x +L+ a n" x n"! Form two polynomials.! Note: a = A), b = B). Bx) = b +b x +b x +L+ b n" x n"! Compute Cx) = Ax)! Bx).! Evaluate C) = a! b.! Running time: On log n) complex arithmetic operations. Theory. [Schönhage-Strassen 97] On log n log log n) bit operations. "the fastest bignum library on the planet" Practice. [GNU ultiple Precision Arithmetic Library] It uses brute force, Karatsuba, and FFT, depending on the size of n. 45