Fast Convolution; Strassen s Method

Transcription

1 Fast Convolution; Strassen s Method 1 Fast Convolution reduction to subquadratic time polynomial evaluation at complex roots of unity interpolation via evaluation at complex roots of unity 2 The Master Method to Solve Recurrences a recursive matrix-matrix multiplication the master method 3 Strassen s method the steps in the method reduction to subcubic time CS 401/MCS 401 Lecture 9 Computer Algorithms I Jan Verschelde, 9 July 2018 Computer Algorithms I (CS 401/MCS 401) Fast Convolution; Strassen s Method L-9 9 July / 40

2 Fast Convolution; Strassen s Method 1 Fast Convolution reduction to subquadratic time polynomial evaluation at complex roots of unity interpolation via evaluation at complex roots of unity 2 The Master Method to Solve Recurrences a recursive matrix-matrix multiplication the master method 3 Strassen s method the steps in the method reduction to subcubic time Computer Algorithms I (CS 401/MCS 401) Fast Convolution; Strassen s Method L-9 9 July / 40

3 computing convolutions Input: a = (a 0, a 1,..., a n 1 ), b = (b 0, b 1,..., b n 1 ). Output: c = a b, the convolution of a with b. Example: n = 4, consider: a 0 b 0 a 0 b 1 a 0 b 2 a 0 b 3 a 1 b 0 a 1 b 1 a 1 b 2 a 1 b 3 a 2 b 0 a 2 b 1 a 2 b 2 a 2 b 3 + a 3 b 0 a 3 b 1 a 3 b 2 a 3 b 3 c 0 c 1 c 2 c 3 c 4 c 5 c 6 The general formula for the k-th coefficient in the convolution: c k = min(n 1,k) i=max(0,k n+1) a k i b i. Computer Algorithms I (CS 401/MCS 401) Fast Convolution; Strassen s Method L-9 9 July / 40

4 applying divide and conquer View the input a = (a 0, a 1,..., a n 1 ) and b = (b 0, b 1,..., b n 1 ) as coefficients of polynomials A(x) and B(x) respectively. Compute c = a b by polynomial evaluation and interpolation: 1 Choose 2n values x 1, x 2,..., x 2n. Evaluating A and B at those values gives A(x 1 ), A(x 2 ),..., A(x 2n ), B(x 1 ), B(x 2 ),..., B(x 2n ). 2 Multiply the 2n evaluations: C(x j ) = A(x j ) B(x j ), j = 1, 2,..., 2n. Evaluation commutes with computing the product of A and B. 3 Because the degree of C is 2n 1, C has 2n 1 coefficients, which can be recovered from the 2n 1 values C(x 1 ), C(x 2 ),..., C(x 2n ) by polynomial interpolation. Steps 2 runs in O(n), but step 1 and 3 are both O(n 2 ), for any x i. Computer Algorithms I (CS 401/MCS 401) Fast Convolution; Strassen s Method L-9 9 July / 40

5 Lagrange interpolation Given are n interpolation points (x i, f i ), i = 0, 1,..., n, where for all i j: x i x j. The Lagrange interpolating polynomial has the form p(x) = l 0 (x)f 0 + l 1 (x)f (x 1 ) + + l n (x)f n, where l i (x j ) = { 1 if i = j 0 if i j l i (x) = n j = 0 j i ( x xj x i x j ). In this form, we have that p(x i ) = f i, i = 0, 1,..., n. Exercise 1: Write a pseudo description for the computation of p(x). Prove that the algorithm to compute p(x) runs in O(n 2 ) time. Computer Algorithms I (CS 401/MCS 401) Fast Convolution; Strassen s Method L-9 9 July / 40

6 Fast Convolution; Strassen s Method 1 Fast Convolution reduction to subquadratic time polynomial evaluation at complex roots of unity interpolation via evaluation at complex roots of unity 2 The Master Method to Solve Recurrences a recursive matrix-matrix multiplication the master method 3 Strassen s method the steps in the method reduction to subcubic time Computer Algorithms I (CS 401/MCS 401) Fast Convolution; Strassen s Method L-9 9 July / 40

7 complex roots of unity Consider the 8-th complex roots of unity: ω 1 = e 2π/8 = cos(2π/8) + sin(2π/8)i, i = 1, ω k = e 2kπ/8, k = 0, 1,..., 7. Let θ = 2π/8: ω 2 ω 3 ω 1 ω 4 θ ω 0 ω 5 ω 6 ω 7 Computer Algorithms I (CS 401/MCS 401) Fast Convolution; Strassen s Method L-9 9 July / 40

8 an exercise Exercise 2: For n > 0, show that the squares of the 2n complex roots of unity are the n complex roots of unity of x n 1. Computer Algorithms I (CS 401/MCS 401) Fast Convolution; Strassen s Method L-9 9 July / 40

9 evaluation at complex roots of unity A(x) = a 0 + a 1 x + a 2 x 2 + a 3 x 3 + a 4 x 4 + a 5 x 5 + a 6 x 6 + a 7 x 7 = a 0 + a 2 x 2 + a 4 x 4 + a 6 x 6 + a 1 x + a 3 x 3 + a 5 x 5 + a 7 x 7 = a 0 + a 2 x 2 + a 4 x 4 + a 6 x 6 + x(a 1 + a 3 x 2 + a 5 x 4 + a 7 x 6 ) A even (z) = a 0 + a 2 z + a 4 z 2 + a 6 z 3 A odd (z) = a 1 + a 3 z + a 5 z 2 + a 7 z 3 We have thus A(x) = A even (x 2 ) + x A odd (x 2 ). Consider: A(ω 0 ) = A even (ω 0 ) + ω 0 A odd (ω 0 ) A(ω 1 ) = A even (ω 2 ) + ω 1 A odd (ω 2 ) A(ω 2 ) = A even (ω 4 ) + ω 2 A odd (ω 4 ) A(ω 3 ) = A even (ω 6 ) + ω 3 A odd (ω 6 ) A(ω 4 ) = A even (ω 0 ) + ω 4 A odd (ω 0 ), as (ω 4 ) 2 = e 2 (2 4π/8) = ω 0 A(ω 5 ) = A even (ω 2 ) + ω 5 A odd (ω 2 ), as (ω 5 ) 2 = e 2 (2 5π/8) = ω 2 A(ω 6 ) = A even (ω 4 ) + ω 6 A odd (ω 4 ), as (ω 6 ) 2 = e 2 (2 6π/8) = ω 4 A(ω 7 ) = A even (ω 6 ) + ω 7 A odd (ω 6 ), as (ω 7 ) 2 = e 2 (2 7π/8) = ω 6 Computer Algorithms I (CS 401/MCS 401) Fast Convolution; Strassen s Method L-9 9 July / 40

10 deriving a recurrence The evaluation of A(x) at 2n points is reduced to 1 evaluation of A even (x) at ω 0, ω 2,..., ω 2n 2, 2 evaluation of A odd (x) at ω 0, ω 2,..., ω 2n 2, 3 n sums of A even (ω 0 ) + ω 0 A odd (ω 0 ) A even (ω 1 ) + ω 1 A odd (ω 1 )..., A even (ω 2n 2 ) + ω n 1 A odd (ω 2n 2 ) 4 n sums of A even (ω 0 ) + ω n A odd (ω 0 ) A even (ω 1 ) + ω n 1 A odd (ω 1 )..., A even (ω 2n 2 ) + ω 2n 1 A odd (ω 2n 2 ) Denote by T (2n) the time to evaluate A(x) at the 2n roots of unity. We have T (2n) 2T (n) + O(n) because 1 the size of A even (x) is n, 2 the size of A odd (x) is n, 3 taking n sums is O(n), 4 taking n sums is O(n). Computer Algorithms I (CS 401/MCS 401) Fast Convolution; Strassen s Method L-9 9 July / 40

11 O(n log(n)) time The recurrence T (2n) 2T (n) + O(n) is equivalent to T (n) 2T (n/2) + O(n). So the following theorem applies: Theorem If T (n) 2T (n/2) + c 1 n and T (2) c 2, for positive constants c 1 and c 2, then T (n) is O(n log(n)). Computer Algorithms I (CS 401/MCS 401) Fast Convolution; Strassen s Method L-9 9 July / 40

12 Fast Convolution; Strassen s Method 1 Fast Convolution reduction to subquadratic time polynomial evaluation at complex roots of unity interpolation via evaluation at complex roots of unity 2 The Master Method to Solve Recurrences a recursive matrix-matrix multiplication the master method 3 Strassen s method the steps in the method reduction to subcubic time Computer Algorithms I (CS 401/MCS 401) Fast Convolution; Strassen s Method L-9 9 July / 40

13 a property of complex roots of unity Consider: ( ) x 2n 1 = (x 1) x 2n x + 1. For any ω, a complex root of unity: ω 2n 1 = 0. Consider ω 1: x 1 0, as ω satisfies x 2n 1 = 0, we must have that the other factor x 2n x + 1 = 0. Because ω 2n 1 = 0, we then have for ω 1: ω 2n ω + 1 = 0. For ω = 1, substitute 1 in ω 2n ω + 1 = 1 } + {{ } = 2n. 2n To conclude: ω 2n ω + 1 = { 0 if ω 1, 2n if ω = 1. Computer Algorithms I (CS 401/MCS 401) Fast Convolution; Strassen s Method L-9 9 July / 40

14 interpolation by evaluation Given C(ω 0 ), C(ω 1 ), C(ω 2 ), C(ω 3 ), C(ω 4 ), C(ω 5 ), C(ω 6 ), C(ω 7 ), we want to compute coefficients c 0, c 1, c 2, c 3, c 4, c 5, c 6, c 7 of C(x) = c 0 + c 1 x + c 2 x 2 + c 3 x 3 + c 4 x 4 + c 5 x 5 + c 6 x 6 + c 7 x 7. D(x) = C(ω 0 )+C(ω 1 )x +C(ω 2 )x 2 +C(ω 3 )x 3 +C(ω 4 )x 4 +C(ω 5 )x 5 +C(ω 6 )x 6 +C(ω 7 )x 7 = (c 0 + c 1 ω 0 + c 2 ω c 3ω c 4ω c 5ω c 0ω c 7ω 7 0 )1 + (c 0 + c 1 ω 1 + c 2 ω c 3ω c 4ω c 5ω c 1ω c 7ω 7 1 )x + (c 0 + c 1 ω 2 + c 2 ω c 3ω c 4ω c 5ω c 2ω c 7ω 7 2 )x 2 + (c 0 + c 1 ω 3 + c 2 ω c 3ω c 4ω c 5ω c 3ω c 7ω 7 3 )x 3 + (c 0 + c 1 ω 4 + c 2 ω c 3ω c 4ω c 5ω c 4ω c 7ω 7 4 )x 4 + (c 0 + c 1 ω 5 + c 2 ω c 3ω c 4ω c 5 ω c 5ω c 7ω 7 5)x 5 + (c 0 + c 1 ω 6 + c 2 ω c 3ω c 4ω c 5ω c 6ω c 7ω 7 6 )x 6 + (c 0 + c 1 ω 7 + c 2 ω c 3 ω c 4ω c 5 ω c 7ω c 7ω 7 7)x 7 Computer Algorithms I (CS 401/MCS 401) Fast Convolution; Strassen s Method L-9 9 July / 40

15 rearranging the coefficients D(x) = (c 0 + c 1 ω 0 + c 2 ω c 3ω c 4ω c 5ω c 0ω c 7ω 7 0 )1 + (c 0 + c 1 ω 1 + c 2 ω c 3ω c 4ω c 5ω c 1ω c 7ω 7 1 )x + (c 0 + c 1 ω 2 + c 2 ω c 3ω c 4ω c 5ω c 2ω c 7ω 7 2 )x 2 + (c 0 + c 1 ω 3 + c 2 ω c 3ω c 4ω c 5ω c 3ω c 7ω 7 3 )x 3 + (c 0 + c 1 ω 4 + c 2 ω c 3ω c 4ω c 5ω c 4ω c 7ω 7 4 )x 4 + (c 0 + c 1 ω 5 + c 2 ω c 3ω c 4ω c 5 ω c 5ω c 7ω 7 5)x 5 + (c 0 + c 1 ω 6 + c 2 ω c 3ω c 4ω c 5ω c 6ω c 7ω 7 6 )x 6 + (c 0 + c 1 ω 7 + c 2 ω c 3 ω c 4ω c 5 ω c 7ω c 7ω 7 7)x 7 = ( 1 + x + x 2 + x 3 + x 4 + x 5 + x 6 + x 7 )c 0 + (ω 0 + ω 1 x + ω 2 x 2 + ω 3 x 3 + ω 4 x 4 + ω 5 x 5 + ω 6 x 6 + ω 7 x 7 )c 1 + (ω0 2 + ω2 1 x + ω2 2 x 2 + ω3 2 x 3 + ω4 2 x 4 + ω5x ω6 2 x 6 + ω7x 2 7 )c 2 + (ω0 3 + ω3 1 x + ω3 2 x 2 + ω3 3 x 3 + ω4 3 x 4 + ω5 3 x 5 + ω6 3 x 6 + ω7 3 x 7 )c 3 + (ω0 4 + ω4 1 x + ω4 2 x 2 + ω3 4 x 3 + ω4 4 x 4 + ω5x ω6 4 x 6 + ω7x 4 7 )c 4 + (ω0 5 + ω5 1 x + ω5 2 x 2 + ω3 5 x 3 + ω4 5 x 4 + ω5 5 x 5 + ω6 5 x 6 + ω7 5 x 7 )c 5 + (ω0 6 + ω6 1 x + ω6 2 x 2 + ω3 6 x 3 + ω4 6 x 4 + ω5 6 x 5 + ω6 6 x 6 + ω7 6 x 7 )c 6 + (ω0 7 + ω7 1 x + ω7 2 x 2 + ω3 7 x 3 + ω4 7 x 4 + ω5x ω6 7 x 6 + ω7x 7 7 )c 7 Computer Algorithms I (CS 401/MCS 401) Fast Convolution; Strassen s Method L-9 9 July / 40

16 simplifying the powers of the roots of unity D(x) = ( 1 + x + x 2 + x 3 + x 4 + x 5 + x 6 + x 7 )c 0 + (ω 0 + ω 1 x + ω 2 x 2 + ω 3 x 3 + ω 4 x 4 + ω 5 x 5 + ω 6 x 6 + ω 7 x 7 )c 1 + (ω0 2 + ω2 1 x + ω2 2 x 2 + ω3 2 x 3 + ω4 2 x 4 + ω5x ω6 2 x 6 + ω7x 2 7 )c 2 + (ω0 3 + ω3 1 x + ω3 2 x 2 + ω3 3 x 3 + ω4 3 x 4 + ω5 3 x 5 + ω6 3 x 6 + ω7 3 x 7 )c 3 + (ω0 4 + ω4 1 x + ω4 2 x 2 + ω3 4 x 3 + ω4 4 x 4 + ω5x ω6 4 x 6 + ω7x 4 7 )c 4 + (ω0 5 + ω5 1 x + ω5 2 x 2 + ω3 5 x 3 + ω4 5 x 4 + ω5 5 x 5 + ω6 5 x 6 + ω7 5 x 7 )c 5 + (ω0 6 + ω6 1 x + ω6 2 x 2 + ω3 6 x 3 + ω4 6 x 4 + ω5 6 x 5 + ω6 6 x 6 + ω7 6 x 7 )c 6 + (ω0 7 + ω7 1 x + ω7 2 x 2 + ω3 7 x 3 + ω4 7 x 4 + ω5x ω6 7 x 6 + ω7x 7 7 )c 7 = (1 + x + x 2 + x 3 + x 4 + x 5 + x 6 + x 7 )c 0 + (1 + ω 1 x + ω 2 x 2 + ω 3 x 3 + ω 4 x 4 + ω 5 x 5 + ω 6 x 6 + ω 7 x 7 )c 1 + (1 + ω 2 x + ω 4 x 2 + ω 6 x 3 + 1x 4 + ω 2 x 5 + ω 4 x 6 + ω 6 x 7 )c 2 + (1 + ω 3 x + ω 6 x 2 + ω 1 x 3 + ω 4 x 4 + ω 7 x 5 + ω 2 x 6 + ω 5 x 7 )c 3 + (1 + ω 4 x + 1x 2 + ω 4 x 3 + 1x 4 + ω 4 x 5 + 1x 6 + ω 4 x 7 )c 4 + (1 + ω 5 x + ω 2 x 2 + ω 7 x 3 + ω 4 x 4 + ω 1 x 5 + ω 6 x 6 + ω 3 x 7 )c 5 + (1 + ω 6 x + ω 4 x 2 + ω 2 x 3 + 1x 4 + ω 6 x 5 + ω 4 x 6 + ω 2 x 7 )c 6 + (1 + ω 7 x + ω 6 x 2 + ω 5 x 3 + ω 4 x 4 + ω 3 x 5 + ω 2 x 6 + ω 1 x 7 )c 7 Computer Algorithms I (CS 401/MCS 401) Fast Convolution; Strassen s Method L-9 9 July / 40

17 simplifying more: ω k = ω1 k, ωj k = ωk j 1 = ω k j mod 2n 1 D(x) = (1 + x + x 2 + x 3 + x 4 + x 5 + x 6 + x 7 )c 0 + (1 + ω 1 x + ω 2 x 2 + ω 3 x 3 + ω 4 x 4 + ω 5 x 5 + ω 6 x 6 + ω 7 x 7 )c 1 + (1 + ω 2 x + ω 4 x 2 + ω 6 x 3 + 1x 4 + ω 2 x 5 + ω 4 x 6 + ω 6 x 7 )c 2 + (1 + ω 3 x + ω 6 x 2 + ω 1 x 3 + ω 4 x 4 + ω 7 x 5 + ω 2 x 6 + ω 5 x 7 )c 3 + (1 + ω 4 x + 1x 2 + ω 4 x 3 + 1x 4 + ω 4 x 5 + 1x 6 + ω 4 x 7 )c 4 + (1 + ω 5 x + ω 2 x 2 + ω 7 x 3 + ω 4 x 4 + ω 1 x 5 + ω 6 x 6 + ω 3 x 7 )c 5 + (1 + ω 6 x + ω 4 x 2 + ω 2 x 3 + 1x 4 + ω 6 x 5 + ω 4 x 6 + ω 2 x 7 )c 6 + (1 + ω 7 x + ω 6 x 2 + ω 5 x 3 + ω 4 x 4 + ω 3 x 5 + ω 2 x 6 + ω 1 x 7 )c 7 = (1 + x + x 2 + x 3 + x 4 + x 5 + x 6 + x 7 )c 0 + (1 + ω 1 x + ω1 2 x 2 + ω1 3 x 3 + ω1 4 x 4 + ω1 5 x 5 + ω1 6 x 6 + ω1 7 x 7 )c 1 + (1 + ω 2 x + ω2 2 x 2 + ω2 3 x 3 + ω2 4 x 4 + ω2 5 x 5 + ω2 6 x 6 + ω2 7 x 7 )c 2 + (1 + ω 3 x + ω3 2 x 2 + ω3 3 x 3 + ω3 4 x 4 + ω3 5 x 5 + ω3 6 x 6 + ω3 7 x 7 )c 3 + (1 + ω 4 x + ω4 2 x 2 + ω4 3 x 3 + ω4 4 x 4 + ω4 5 x 5 + ω4 6 x 6 + ω4 7 x 7 )c 4 + (1 + ω 5 x + ω5x ω5 3 x 3 + ω5x ω5 5 x 5 + ω5 6 x 6 + ω5x 7 7 )c 5 + (1 + ω 6 x + ω6 2 x 2 + ω6 3 x 3 + ω6 4 x 4 + ω6 5 x 5 + ω6 6 x 6 + ω6 7 x 7 )c 6 + (1 + ω 7 x + ω7x ω7 3 x 3 + ω7x ω7 5 x 5 + ω7 6 x 6 + ω7x 7 7 )c 7 Computer Algorithms I (CS 401/MCS 401) Fast Convolution; Strassen s Method L-9 9 July / 40

18 evaluation at the roots of unity D(x) = (1 + x + x 2 + x 3 + x 4 + x 5 + x 6 + x 7 )c 0 + (1 + ω 1 x + ω1 2 x 2 + ω1 3 x 3 + ω1 4 x 4 + ω1 5 x 5 + ω1 6 x 6 + ω1 7 x 7 )c 1 + (1 + ω 2 x + ω2 2 x 2 + ω2 3 x 3 + ω2 4 x 4 + ω2 5 x 5 + ω2 6 x 6 + ω2 7 x 7 )c 2 + (1 + ω 3 x + ω3 2 x 2 + ω3 3 x 3 + ω3 4 x 4 + ω3 5 x 5 + ω3 6 x 6 + ω3 7 x 7 )c 3 + (1 + ω 4 x + ω4 2 x 2 + ω4 3 x 3 + ω4 4 x 4 + ω4 5 x 5 + ω4 6 x 6 + ω4 7 x 7 )c 4 + (1 + ω 5 x + ω5x ω5 3 x 3 + ω5x ω5 5 x 5 + ω5 6 x 6 + ω5x 7 7 )c 5 + (1 + ω 6 x + ω6 2 x 2 + ω6 3 x 3 + ω6 4 x 4 + ω6 5 x 5 + ω6 6 x 6 + ω6 7 x 7 )c 6 + (1 + ω 7 x + ω7x ω7 3 x 3 + ω7x ω7 5 x 5 + ω7 6 x 6 + ω7x 7 7 )c 7 { ω 2n 1 0 if ω 1, + + ω + 1 = 2n if ω = 1, D(ω 0 ) = 8c 0 D(ω 1 ) = 8c 7 D(ω 2 ) = 8c 6 D(ω implies 3 ) = 8c 5 D(ω 4 ) = 8c 4 D(ω 5 ) = 8c 3 D(ω 6 ) = 8c 2 D(ω 7 ) = 8c 1 Computer Algorithms I (CS 401/MCS 401) Fast Convolution; Strassen s Method L-9 9 July / 40

19 the general form of interpolation by evaluation Let D(x) = Then D(ω j ) = 2n 1 k=0 2n 1 C(ω k )x k = k=0 2n 1 l=0 ( 2n 1 l=0 c l ω l k 2n 1 k=0 ) ( 2n 1 c l ω l k l=0 2n 1 ω k j = l=0 ) x k. ( 2n 1 c l k=0 2n 1 ( 2n 1 ) and D(ω j ) = c l e (2πi/2n)(k l+j k) = k=0 l=0 ω l k ωk j ), ( 2n 1 c l k=0 ω l+j k ). ω 2n ω + 1 = { 0 if ω 1, 2n if ω = 1, implies D(ω k ) = 2n c 2n k, for k = 0, 1,..., 2n 1. Or equivalently, c k = D(ω 2n k). 2n Computer Algorithms I (CS 401/MCS 401) Fast Convolution; Strassen s Method L-9 9 July / 40

20 a recursive step 3 For D(x) = 2n 1 k=0 C(ω k )x k, we have c k = D(ω 2n k), k = 0, 1,..., 2n 1. 2n We evaluate a polynomial at complex roots of unity, so the same recurrence T (2n) T (n) + O(n) applies and we obtain a subquadratic, O(n log(n)) algorithm to compute the coefficients of the convolution. Theorem For a = (a 0, a 1,..., a n 1 ) and b = (b 0, b 1,..., b n 1 ), the convolution c = a b can be computed in O(n log(n)) time. The algorithm using the complex roots of unity is known as the Fast Fourier Transform, abbreviated as FFT. Computer Algorithms I (CS 401/MCS 401) Fast Convolution; Strassen s Method L-9 9 July / 40

21 reformulate step 3 with matrices Exercise 3: The computation of the coefficients of C on slide #17 from the values C(x 0 ), C(x 1 ),..., C(x 2n 1 ) can be expressed as a matrix-vector product: 1 The input of this product are the coefficients of C, and 2 the output are the values C(x 0 ), C(x 1 ),..., C(x 2n 1 ). Write the matrix and its inverse for n = 4 and n = 8. Does the structure of the matrix show the reduction to subquadratic time? Computer Algorithms I (CS 401/MCS 401) Fast Convolution; Strassen s Method L-9 9 July / 40

22 Fast Convolution; Strassen s Method 1 Fast Convolution reduction to subquadratic time polynomial evaluation at complex roots of unity interpolation via evaluation at complex roots of unity 2 The Master Method to Solve Recurrences a recursive matrix-matrix multiplication the master method 3 Strassen s method the steps in the method reduction to subcubic time Computer Algorithms I (CS 401/MCS 401) Fast Convolution; Strassen s Method L-9 9 July / 40

23 a recursive matrix-matrix multiplication Given are two n-by-n matrices A and B, we want to compute their product C = A B. Partition each matrix into four n/2-by-n/2-matrices: [ ] [ ] [ A11 A A = 12 B11 B, B = 12 C11 C, C = 12 A 21 A 22 B 21 B 22 C 21 C 22 ]. Then the product of A and B is [ ] [ C11 C 12 A11 B = 11 + A 12 B 21 A 11 B 12 + A 12 B 22 C 21 C 22 A 21 B 11 + A 22 B 21 A 21 B 12 + A 22 B 22 ]. One matrix multiplication A B on an n-by-n matrix is replaced by 8 matrix multiplications on n/2-by-n/2 matrices, and 4 additions of n/2-by-n/2 matrices. Computer Algorithms I (CS 401/MCS 401) Fast Convolution; Strassen s Method L-9 9 July / 40

24 setting up the recurrence Let T (n) be the time to multiply two n-by-n matrices. The base case for the recursion is n = 1: c 11 = a 11 b 11, which runs in Θ(1), constant time. For n > 1: [ C11 C 12 C 21 C 22 ] [ A11 B = 11 + A 12 B 21 A 11 B 12 + A 12 B 22 A 21 B 11 + A 22 B 21 A 21 B 12 + A 22 B 22 ]. The addition of two matrices of dimension n costs Θ(n 2 ). Thus, we have the recurrence: T (n) = 8T (n/2) + Θ(n 2 ), for n > 1. Computer Algorithms I (CS 401/MCS 401) Fast Convolution; Strassen s Method L-9 9 July / 40

25 Fast Convolution; Strassen s Method 1 Fast Convolution reduction to subquadratic time polynomial evaluation at complex roots of unity interpolation via evaluation at complex roots of unity 2 The Master Method to Solve Recurrences a recursive matrix-matrix multiplication the master method 3 Strassen s method the steps in the method reduction to subcubic time Computer Algorithms I (CS 401/MCS 401) Fast Convolution; Strassen s Method L-9 9 July / 40

26 Theorem (the master theorem) The recurrence relation T (n) = at (n/b) + f (n), a 1, b > 1, some function f has the following bounds: 1 If f (n) is O(n log b (a) ɛ ) for some constant ɛ > 0, then T (n) is Θ(n log b (a) ). 2 If f (n) is Θ(n log b (a) ), then T (n) is Θ(n log b (a) log 2 (n)). 3 If f (n) is Ω(n log b (a)+ɛ ), for some constant ɛ > 0, and a f (n/b) c f (n), for some constant c < 1 and n N, then T (n) is Θ(f (n)). Computer Algorithms I (CS 401/MCS 401) Fast Convolution; Strassen s Method L-9 9 July / 40

27 interpretation of the theorem Recall: the recurrence T (n) = at (n/b) + f (n) defines a tree of depth log b (n) with L = n log b (a) leaves. Three cases: 1 f grows no faster than L: f is O(n log b (a) ) 2 f grows at the same rate as L: f is Θ(n log b (a) ) 3 f grows at least as fast as L: f is Ω(n log b (a) ) Relating f (n) to L, we find that T (n) depends on L. Second case: f is Θ(L) T (n) is Θ(L log 2 (n)). Note: depth log b (n) = log 2 (n)/ log 2 (b) is Θ(log 2 (n)). Computer Algorithms I (CS 401/MCS 401) Fast Convolution; Strassen s Method L-9 9 July / 40

28 an illustration: merge sort The merge sort algorithm has a recurrence T (n) = 2T (n/2) + Θ(n), where a = 2, b = 2, and f (n) = Θ(n). The number of leaves: L = n log b (a) = n log 2 (2) = n, so case 2 of the theorem applies: 2 If f (n) is Θ(n log b (a) ), then T (n) is Θ(n log b (a) log 2 (n)). Therefore: T (n) is O(n log 2 (n)). Computer Algorithms I (CS 401/MCS 401) Fast Convolution; Strassen s Method L-9 9 July / 40

29 applied to recursive matrix-matrix multiplication The recursive matrix-matrix multiplication has the recurrence T (n) = 8T (n/2) + Θ(n 2 ), where a = 8, b = 2, and f (n) = Θ(n 2 ). Compute n log b (a) = n log 2 (8) = n 3, so case 1 of the theorem applies: 1 If f (n) is O(n log b (a) ɛ ) for some constant ɛ > 0, then T (n) is Θ(n log b (a) ). Take ɛ = 1. Therefore: T (n) is O(n 3 ). Computer Algorithms I (CS 401/MCS 401) Fast Convolution; Strassen s Method L-9 9 July / 40

30 applying the master method Exercise 4: Use the master method to give a tight asymptotic bound on the recurrence T (n) = 2T (n/4) + n. Computer Algorithms I (CS 401/MCS 401) Fast Convolution; Strassen s Method L-9 9 July / 40

31 Fast Convolution; Strassen s Method 1 Fast Convolution reduction to subquadratic time polynomial evaluation at complex roots of unity interpolation via evaluation at complex roots of unity 2 The Master Method to Solve Recurrences a recursive matrix-matrix multiplication the master method 3 Strassen s method the steps in the method reduction to subcubic time Computer Algorithms I (CS 401/MCS 401) Fast Convolution; Strassen s Method L-9 9 July / 40

32 steps 1 and 2 of Strassen s method Given are two n-by-n matrices A and B, we want to compute their product C = A B. 1 Partition each matrix into four n/2-by-n/2-matrices: ] [ C11 C, C = 12 C 21 C 22 [ A11 A A = 12 A 21 A 22 ] [ B11 B, B = 12 B 21 B 22 2 Make the following 10 matrices S 1 = B 12 B 22, S 2 = A 11 + A 12, S 3 = A 21 + A 22, S 4 = B 21 B 11, S 5 = A 11 + A 22, S 6 = B 11 + B 22, S 7 = A 12 A 22, S 8 = B 21 + B 22, S 9 = A 11 A 21, S 10 = B 11 + B 12. This steps takes Θ(n 2 ) time. ]. Computer Algorithms I (CS 401/MCS 401) Fast Convolution; Strassen s Method L-9 9 July / 40

33 steps 3 and 4 of Strassen s method 3 Multiply seven times n/2-by-n/2 matrices: P 1 = A 11 S 1 = A 11 B 12 A 11 B 22, P 2 = S 2 B 22 = A 11 B 22 + A 12 B 22, P 3 = S 3 B 11 = A 21 B 11 + A 22 B 11, P 4 = A 22 S 4 = A 22 B 21 A 22 B 11, P 5 = S 5 S 6 = A 11 B 11 + A 11 B 22 + A 22 B 11 + A 22 B 22, P 6 = S 7 S 8 = A 12 B 21 + A 12 B 22 A 22 B 21 A 22 B 22, P 7 = S 9 S 10 = A 11 B 11 + A 11 B 12 A 12 B 11 A 21 B Construct the matrices in the product C: C 11 = P 5 + P 4 P 2 + P 6, C 12 = P 1 + P 2, C 21 = P 3 + P 4, C 22 = P 5 + P 1 P 3 P 7. Computer Algorithms I (CS 401/MCS 401) Fast Convolution; Strassen s Method L-9 9 July / 40

34 verifying correctness for C 12 Strassen s method defines C 12 = P 1 + P 2, with With direct verication: P 1 = A 11 S 1 = A 11 B 12 A 11 B 22, P 2 = S 2 B 22 = A 11 B 22 + A 12 B 22, A 11 B 12 A 11 B 22 + A 11 B 22 + A 12 B 22 A 11 B 12 + A 12 B 22 Indeed, we computed earlier: C 12 = A 11 B 12 + A 12 B 22. Computer Algorithms I (CS 401/MCS 401) Fast Convolution; Strassen s Method L-9 9 July / 40

35 verifying correctness for C 21 Strassen s method defines C 21 = P 3 + P 4, with With direct verification: P 3 = S 3 B 11 = A 21 B 11 + A 22 B 11, P 4 = A 22 S 4 = A 22 B 21 A 22 B 11, A 21 B 11 + A 22 B 11 + A 22 B 11 + A 22 B 21 A 21 B 11 + A 22 B 21 Indeed, we computed earlier: C 21 = A 21 B 11 + A 22 B 21. Computer Algorithms I (CS 401/MCS 401) Fast Convolution; Strassen s Method L-9 9 July / 40

36 verifying correctness for C 11 Strassen s method defines C 11 = P 5 + P 4 P 2 + P 6, with P 5 = A 11 B 11 + A 11 B 22 + A 22 B 11 + A 22 B 22, P 4 = A 22 B 21 A 22 B 11, P 2 = A 11 B 22 + A 12 B 22, P 6 = A 12 B 21 + A 12 B 22 A 22 B 21 A 22 B 22. With direct verification: A 11 B 11 +A 11 B 22 +A 22 B 11 +A 22 B 22 A 22 B 11 +A 22 B 21 A 11 B 22 +A 12 B 22 + A 22 B 22 A 22 B 21 +A 12 B 22 +A 12 B 21 A 11 B 11 +A 12 B 21 Indeed, we computed earlier: C 11 = A 11 B 11 + A 12 B 21. Computer Algorithms I (CS 401/MCS 401) Fast Convolution; Strassen s Method L-9 9 July / 40

37 verifying correctness for C 22 Strassen s method defines C 22 = P 5 + P 1 P 3 P 7, with P 5 = A 11 B 11 + A 11 B 22 + A 22 B 11 + A 22 B 22, P 1 = A 11 B 12 A 11 B 22, P 3 = A 21 B 11 + A 22 B 11, P 7 = A 11 B 11 + A 11 B 12 A 12 B 11 A 21 B 12. With direct verification: A 11 B 11 +A 11 B 22 +A 22 B 11 +A 22 B 22 A 11 B 22 +A 11 B 12 A 22 B 11 A 21 B 11 + A 11 B 11 A 11 B 12 +A 21 B 11 +A 21 B 12 A 22 B 22 +A 21 B 12 Indeed, we computed earlier: C 22 = A 21 B 12 + A 22 B 22. Computer Algorithms I (CS 401/MCS 401) Fast Convolution; Strassen s Method L-9 9 July / 40

38 Fast Convolution; Strassen s Method 1 Fast Convolution reduction to subquadratic time polynomial evaluation at complex roots of unity interpolation via evaluation at complex roots of unity 2 The Master Method to Solve Recurrences a recursive matrix-matrix multiplication the master method 3 Strassen s method the steps in the method reduction to subcubic time Computer Algorithms I (CS 401/MCS 401) Fast Convolution; Strassen s Method L-9 9 July / 40

39 reduction to subcubic time Let T (n) be the time of Strassen s method. We have the recurrence { θ(1) if n = 1, T (n) = 7T (n/2) + θ(n 2 ) if n > 1. where a = 7, b = 2, and f (n) = Θ(n 2 ). Compute n log b (a) = n log 2 (7), so case 1 of the theorem applies: 1 If f (n) is O(n log b (a) ɛ ) for some constant ɛ > 0, then T (n) is Θ(n log b (a) ). Take ɛ = 0.80, as log 2 (7) Therefore: T (n) is O(n log 2 (7) ) = Θ(n ). Computer Algorithms I (CS 401/MCS 401) Fast Convolution; Strassen s Method L-9 9 July / 40

40 exercises Homework collection on Friday 13 July 2018, at 10AM: 1 Exercise 1 on slide #10 of lecture 7. 2 Exercise 2 on slide #2 of lecture 7. 3 Exercise 1 on slide #9 of lecture 8. 4 Exercise 2 on slide #16 of lecture 8. 5 Exercise 3 on slide #16 of lecture 8. 6 Exercise 4 on slide #28 of lecture 8. 7 Exercise 1 on slide #5 of lecture 9. 8 Exercise 2 on slide #8 of lecture 9. 9 Exercise 3 on slide #20 of lecture Exercise 4 on slide #29 of lecture 9. Computer Algorithms I (CS 401/MCS 401) Fast Convolution; Strassen s Method L-9 9 July / 40