Chapter 1 Divide and Conquer Polynomial Multiplication Algorithm Theory WS 2015/16 Fabian Kuhn

Chapter 1 Divide and Conquer Polynomial Multiplication Algorithm Theory WS 2015/16 Fabian Kuhn

Formulation of the D&C principle Divide-and-conquer method for solving a problem instance of size n: 1. Divide n c: Solve the problem directly. n > c: Divide the problem into k subproblems of sizes n 1,, n k < n (k 2). 2. Conquer Solve the k subproblems in the same way (recursively). 3. Combine Combine the partial solutions to generate a solution for the original instance. Algorithm Theory, WS 2015/16 Fabian Kuhn 2

Analysis Recurrence relation: T(n) : max. number of steps necessary for solving an instance of size n T(n) = a if n c T n 1 + + T n k if n > c + cost for divide and combine Special case: k = 2, n 1 = n 2 = n 2 cost for divide and combine: DC n T(1) = a T(n) = 2T(n/2) + DC(n) Algorithm Theory, WS 2015/16 Fabian Kuhn 3

Recurrence Relations: Master Theorem Recurrence relation T n = a T n b + f n, T n = O 1 for n n 0 Cases f n = O(n c ), c < log b a T n = Θ n log b a f n = Ω n c, c > log b a T n = Θ f n f n = Θ n c log k n, c = log b a T n = Θ n c log k+1 n Algorithm Theory, WS 2015/16 Fabian Kuhn 4

Polynomials Real polynomial p in one variable x: p x = a n x n + a n 1 x n 1 + + a 1 x 1 + a 0 Coefficients of p: a 0, a 1,, a n R Degree of p: largest power of x in p (n in the above case) Example: p(x) = 3x 3 15x 2 + 18x Set of all real-valued polynomials in x: R[x] (polynomial ring) Algorithm Theory, WS 2015/16 Fabian Kuhn 5

Operations: Addition Given: Polynomials p, q R[x] of degree n p x = a n x n + a n 1 x n 1 + + a 1 x + a 0 q x = b n x n + b n 1 x n 1 + + b 1 x + b 0 Compute sum p x + q x : p x + q x = a n x n + + a 0 + b n x n + + b 0 = a n + b n x n + + a 1 + b 1 x + (a 0 + b 0 ) Algorithm Theory, WS 2015/16 Fabian Kuhn 6

Operations: Multiplication Given: Polynomials p, q R[x] of degree n p x = a n x n + a n 1 x n 1 + + a 1 x + a 0 q x = b n x n + b n 1 x n 1 + + b 1 x + b 0 Product p x q(x): p x q x = a n x n + + a 0 b n x n + + b 0 = c 2n x 2n + c 2n 1 x 2n 1 + + c 1 x + c 0 Obtaining c i : what products of monomials have degree i? For 0 i 2n: c i = a j b i j where a i = b i = 0 for i > n. i j=0 Algorithm Theory, WS 2015/16 Fabian Kuhn 7

Operations: Evaluation Given: Polynomial p R[x] of degree n p x = a n x n + a n 1 x n 1 + + a 1 x + a 0 Horner s method for evaluation at specific value x 0 : p x 0 = a n x 0 + a n 1 x 0 + a n 2 x 0 + + a 1 x 0 + a 0 Pseudo-code: p a n ; i n; while (i > 0) do i i 1; p p x 0 + a i end Running time: O(n) Algorithm Theory, WS 2015/16 Fabian Kuhn 8

Representation of Polynomials Coefficient representation: Polynomial p x R[x] of degree n is given by its n + 1 coefficients a 0,, a n : p x = a n x n + + a 1 x + a 0 Example: p x = 3x 3 15x 2 + 18x The most typical (and probably most natural) representation of polynomials Algorithm Theory, WS 2015/16 Fabian Kuhn 9

Representation of Polynomials Product of linear factors: Polynomial p x C[x]of degree n is given by its n roots p x = a n x x 1 x x 2 (x x n ) Example: p x = 3x x 2 x 3 Every polynomial has exactly n roots x i C for which p x i = 0 Polynomial is uniquely defined by the n roots and a n We will not use this representation Algorithm Theory, WS 2015/16 Fabian Kuhn 10

Representation of Polynomials Point-value representation: Polynomial p x R[x] of degree n is given by n + 1 point-value pairs: p = x 0, p x 0, x 1, p x 1,, x n, p x n where x i x j for i j. Example: The polynomial p x = 3x x 2 x 3 is uniquely defined by the four point-value pairs 0,0, 1,6, 2,0, 3,0. Algorithm Theory, WS 2015/16 Fabian Kuhn 11

Operations: Coefficient Representation Deg.-n polynomials p x = a n x n + + a 0, q x = b n x n + + b 0 Addition: p x + q x = a n + b n x n + + (a 0 + b 0 ) Time: O(n) Multiplication: i p x q x = c 2n x 2n + + c 0, where c i = a j b i j j=0 Naive solution: Need to compute product a i b j for all 0 i, j n Time: O(n 2 ) Algorithm Theory, WS 2015/16 Fabian Kuhn 12

Operations Point-Value Representation Degree-n polynomials p = x 0, p x 0,, x n, p x n, q = x 0, q x 0,, x n, q x n Note: we use the same points x 0,, x n for both polynomials Addition: p + q = x 0, p x 0 + q x 0,, x n, p x n + q x n Time: O(n) Multiplication: p q = x 0, p x 0 q x 0,, x 2n, p x 2n q x 2n Time: O(n) Algorithm Theory, WS 2015/16 Fabian Kuhn 13

Faster Multiplication? Multiplication is slow Θ n 2 when using the standard coefficient representation Try divide-and-conquer to get a faster algorithm Assume: degree is n 1, n is even Divide polynomial p x = a n 1 x n 1 + + a 0 into 2 polynomials of degree n 2 1: p 0 x = an 2 1x n 2 1 + + a 0 p 1 x = a n 1 x n 2 1 + + an 2 p x = p 1 x x n 2 + p 0 x Similarly: q x = q 1 x x n 2 + q 0 x Algorithm Theory, WS 2015/16 Fabian Kuhn 14

Use Divide-And-Conquer Divide: p x = p 1 x x n 2 + p 0 x, q x = q 1 x x n 2 + q 0 x Multiplication: p x q x = p 1 x q 1 x x n + p 0 x q 1 x + p 1 x q 0 (x) x n 2 + p 0 x q 0 (x) 4 multiplications of degree n 2 1 polynomials: T n = 4T n 2 + O n Leads to T n = Θ n 2 like the naive algorithm follows immediately by using the master theorem Algorithm Theory, WS 2015/16 Fabian Kuhn 15

More Clever Recursive Solution Recall that p x q x = p 1 x q 1 x x n + p 0 x q 1 x + p 1 x q 0 (x) x n 2 + p 0 x q 0 (x) Compute r x = p 0 x + p 1 x q 0 x + q 1 x : Algorithm Theory, WS 2015/16 Fabian Kuhn 16

Karatsuba Algorithm Recursive multiplication: r x = p 0 x + p 1 x q 0 x + q 1 x p x q x = p 1 x q 1 x x n + r x p 0 x q 0 x + p 1 x q 1 (x) x n 2 + p 0 x q 0 (x) Recursively do 3 multiplications of degr. n 2 1 -polynomials T n = 3T n 2 + O(n) Gives: T n = O n 1.59 (see Master theorem) Algorithm Theory, WS 2015/16 Fabian Kuhn 17

Faster Polynomial Multiplication? Multiplication is fast when using the point-value representation Idea to compute p x q(x) (for polynomials of degree < n): p, q of degree n 1, n coefficients Evaluation at points x 0, x 1,, x 2n 1 2 2n point-value pairs x i, p x i and x i, q x i Point-wise multiplication 2n point-value pairs x i, p x i q x i Interpolation p x q(x) of degree 2n 2, 2n 1 coefficients Algorithm Theory, WS 2015/16 Fabian Kuhn 18

Reminder Complex Numbers Algorithm Theory, WS 2015/16 Fabian Kuhn 19

Point-Value Representation of p, q Select points x 0, x 1,, x N 1 to evaluate p and q in a clever way Consider the N powers of the principle Nth root of unity: Principle root of unity: ω N = e 2πi N i = 1, e 2πi = 1 ω 8 3 ω 8 2 ω 8 1 Powers of ω N (roots of unity): ω 8 4 ω 8 0 = 1 1 = ω N 0, ω N 1,, ω N N 1 ω 8 5 ω 8 6 ω 8 7 Note: ω N k = e 2πik N = cos 2πk N + i sin 2πk N Algorithm Theory, WS 2015/16 Fabian Kuhn 20

Discrete Fourier Transform The values p ω N i for i = 0,, N 1 uniquely define a polynomial p of degree < N. Discrete Fourier Transform (DFT): Assume a = a 0,, a N 1 is the coefficient vector of poly. p p x = a N 1 x N 1 + + a 1 x + a 0 DFT N a p ω N 0, p ω N 1,, p ω N N 1 Algorithm Theory, WS 2015/16 Fabian Kuhn 21

Example Consider polynomial p x = 3x 3 15x 2 + 18x Choose N = 4 Roots of unity: ω 4 1 = i ω 4 2 = 1 ω 4 0 = 1 ω 4 3 = i Algorithm Theory, WS 2015/16 Fabian Kuhn 22

Example Consider polynomial p x = 3x 3 15x 2 + 18x N = 4, roots of unity: ω 4 0 = 1, ω 4 1 = i, ω 4 2 = 1, ω 4 3 = i Evaluate p(x) at ω 4 k : ω 4 0, p ω 4 0 = 1, p 1 = 1,6 ω 4 1, p ω 4 1 = i, p i = i, 15 + 15i ω 4 2, p ω 4 2 = 1, p 1 = 1, 36 ω 4 3, p ω 4 3 = i, p i = i, 15 15i For a = 0,18, 15,3 : DFT 4 a = (6, 15 + 15i, 36, 15 15i) Algorithm Theory, WS 2015/16 Fabian Kuhn 23

Faster Polynomial Multiplication? Idea to compute p x q(x) (for polynomials of degree < n): p, q of degree n 1, n coefficients Evaluation at points ω 0 2n, ω 1 2n 1 2n,, ω 2n 2 2n point-value pairs ω k 2n, p ω 2n Point-wise multiplication k and ω k 2n k, q ω 2n 2n point-value pairs ω k k 2n, p ω 2n k q ω 2n Interpolation p x q(x) of degree 2n 2, 2n 1 coefficients Algorithm Theory, WS 2015/16 Fabian Kuhn 24

Properties of the Roots of Unity Cancellation Lemma: For all integers n > 0, k 0, and d > 0, we have: Proof: ω dk dn = ωk n, ω k+n n = ωk n Algorithm Theory, WS 2015/16 Fabian Kuhn 25

Divide-and-Conquer Approach Divide p x of degree N 1 (N is even) into 2 polynomials of degree N 2 1 differently than in Karatsuba s algorithm p 0 x = a 0 + a 2 x + a 4 x 2 + + a N 2 x N 2 1 (even coeff.) p 1 x = a 1 + a 3 x + a 5 x 2 + + a N 1 x N 2 1 (odd coeff.) Algorithm Theory, WS 2015/16 Fabian Kuhn 26

Discrete Fourier Transform Evaluation for k = 0,, N 1: p ω N k = p 0 (ω N k ) 2 + ω N k p 1 (ω N k ) 2 = p k 0 ω N 2 p 0 ω N 2 + ω k k N p 1 ω N 2 k N 2 + ω N k p 1 ω N 2 k N 2 if k < N 2 if k N 2 For the coefficient vector a of p x : 0 DFT N a = p 0 ω N 2 + ω 0 0 N p 1 ω N 2 N,, p 0 ω 2 1 0 N 2, p 0 ω N 2 N 2 1,, p 0 ω N 2 N,, ω 2 1 N N p 1 ω 2 1 N N 2, ω 2 0 N p 1 ω N 2,, ω N 1 N 2 1 N p 1 ω N 2 Algorithm Theory, WS 2015/16 Fabian Kuhn 27

Example For the coefficient vector a of p x : 0 DFT N a = p 0 ω N 2 N,, p 0 ω 2 1 0 N 2, p 0 ω N 2 N 2 1,, p 0 ω N 2 + ω 0 0 N p 1 ω N 2 N,, ω 2 1 N N p 1 ω 2 1 N N 2, ω 2 0 N p 1 ω N 2,, ω N 1 N 2 1 N p 1 ω N 2 N = 4: p ω 4 0 = p 0 ω 2 0 + ω 4 0 p 1 ω 2 0 p ω 4 1 = p 0 ω 2 1 + ω 4 1 p 1 ω 2 1 p ω 4 2 = p 0 ω 2 0 + ω 4 2 p 1 ω 2 0 p ω 4 3 = p 0 ω 2 1 + ω 4 3 p 1 ω 2 1 Need: p 0 ω 2 0, p 0 ω 2 1 and p 1 ω 2 0, p 1 ω 2 1 (DFTs of coefficient vectors of p 0 and p 1 ) Algorithm Theory, WS 2015/16 Fabian Kuhn 28

Recursive Structure For simplicity, we abuse notation in the following: Poly. p x = a N 1 x N 1 + + a 0 with coefficient vector a Let DFT N p DFT N a Recursive structure: For N = 4: DFT 4 p k = p ω 4 k = DFT 2 p 0 k mod 2 + ω 4 k DFT 2 p 1 k mod 2 General N (assume N is even): DFT N p k = p ω N k = DFT N/2 p 0 k mod N 2 + ω N k DFT N/2 p 1 k mod N 2 Algorithm Theory, WS 2015/16 Fabian Kuhn 29

Computation of DFT N Divide-and-conquer algorithm for DFT N (p): 1. Divide N 1: DFT 1 p = a 0 N > 1: Divide p into p 0 (even coeff.) and p 1 (odd coeff). 2. Conquer Solve DFT N 2 (p 0 ) and DFT N 2 (p 1 ) recursively 3. Combine Compute DFT N (p) based on DFT N 2 (p 0 ) and DFT N 2 p 1 Algorithm Theory, WS 2015/16 Fabian Kuhn 30

Analysis T N : time to compute DFT N (p): T N = 2T N 2 + O N, T 1 = O(1) As for mergesort, comparing orders, closest pair of points: T N = O(N log N) Algorithm Theory, WS 2015/16 Fabian Kuhn 31

Small Improvement Polynomial p of degree N 1: p ω k N = p k 0 ω N 2 p 0 ω N 2 = p k 0 ω N 2 p 0 ω N 2 + ω k k N p 1 ω N 2 k N 2 + ω N k p 1 ω N 2 + ω k k N p 1 ω N 2 k N 2 k N 2 ω N k N 2 p 1 ω N 2 k N 2 if k < N 2 if k N 2 if k < N 2 if k N 2 k Need to compute p 0 ω N 2 and ω k k N p 1 ω N 2 for 0 k < N 2. Algorithm Theory, WS 2015/16 Fabian Kuhn 32

Example p ω 0 4 = p 0 ω 0 2 + ω 0 0 4 p 1 ω 2 p ω 1 4 = p 0 ω 1 2 + ω 1 4 p 1 (ω 1 2 ) p ω 2 4 = p 0 ω 0 2 ω 0 4 p 1 (ω 0 2 ) p ω 3 4 = p 0 ω 1 2 ω 1 4 p 1 (ω 1 2 ) Algorithm Theory, WS 2015/16 Fabian Kuhn 33

Fast Fourier Transform (FFT) Algorithm Algorithm FFT(a) Input: Array a of length N, where N is a power of 2 Output: DFT N a if n = 1 then return a 0 ; // a = a 0 d 0 FFT a 0, a 2,, a N 2 ; d 1 FFT a 1, a 3,, a N 1 ; ω N e 2πi N ; ω 1; for k = 0 to N 2 1 do k // ω = ω N x ω d k 1 ; d k d k 0 + x; d k+n 2 d k 0 x; ω ω ω N end; return d = [d 0, d 1,, d N 1 ]; Algorithm Theory, WS 2015/16 Fabian Kuhn 34