The Master Theorem for solving recurrences lgorithms and Data Structures Strassen s lgorithm 23rd September, 2014 Theorem Let n 0 N, k N 0 and a, b R with a > 0 and b > 1, and let T : N R satisfy the following recurrence: { Θ(1) if n < n 0, T (n) a T (n/b) + Θ(n k ) if n n 0. Let c log b (a); we call c the critical exponent. Then Θ(n c ) if k < c (I), T (n) Θ(n c lg(n)) if k c (II), Θ(n k ) if k > c (III). Theorem also holds if we replace a T (n/b) above by a 1 T ( n/b ) + a 2 T ( n/b ) for any a 1, a 2 0 with a 1 + a 2 a. Start next week (week 3) Tutorials DS (2014/15) Lecture 3 slide 1 Tutorial allocations will soon appear on the course webpage http://www.inf.ed.ac.uk/teaching/courses/ads/ The Master Theorem (cont d) DS (2014/15) Lecture 3 slide 3 We don t have time to prove the Master Theorem in class. You can find the proof in Section 4.6 of [CLRS]. Section 4.4 of [CLRS], 2nd ed. Their version of the M.T. is a bit more general than ours. Consider the following examples: T (n) 4T (n/2) + n, T (n) 4T ( n/2 ) + n 2, T (n) 4T (n/2) + n 3. Could alternatively unfold-and-sum to guess, then prove, the first and third of these. CLSS EXERCISE DS (2014/15) Lecture 3 slide 2 DS (2014/15) Lecture 3 slide 4
Recall Matrix Multiplication The product of two (n n)-matrices (a ij ) 1 i,j n and (b ij ) 1 i,j n is the (n n)-matrix C where C (c ij ) 1 i,j n with entries n c ij a ik b kj. straightforward algorithm lgorithm MatMult(, ) 1. n number of rows of 2. for i 1 to n do 3. for j 1 to n do 4. c ij 0 5. for k 1 to n do 6. c ij c ij + a ik b kj 7. return C (c ij ) 1 i,j n The Matrix Multiplication Problem Input: (n n)-matrices and Output: the (n n)-matrix Requires Θ(n 3 ) arithmetic operations (additions and multiplications). row i c ij Matrix Multiplication column j - n multiplications and n additions for each c ij. - there are n 2 different c ij entries. DS (2014/15) Lecture 3 slide 5 b 1j b2j a i1 a i2 a in b nj DS (2014/15) Lecture 3 slide 7 näive divide-and-conquer algorithm Observe If 11 12 11 12 and 21 22 21 22 for (n/2 n/2)-submatrices ij and ij then 11 11 + 12 21 11 12 + 12 22 21 11 + 22 21 21 12 + 22 22 note: We are assuming n is a power of 2. DS (2014/15) Lecture 3 slide 6 DS (2014/15) Lecture 3 slide 8
row i näive divide-and-conquer algorithm c ij column j 11 12 a i1 a i2 21 22 a in b b 1j 2j 11 12 21 22 b nj Suppose i n/2 and j > n/2. Then n/2 n n c ij a ik b kj a ik b kj + a ik b kj kn/2+1 11 12 12 22 nalysis of D&C-MatMult T (n) is the number of operations done by D&C-MatMult. Lines 1, 2, 3, 4, 7 require Θ(1) arithmetic operations Line 5 requires 8T (n/2) arithmetic operations Line 6 requires 4(n/2) 2 Θ(n 2 ) arithmetic operations. Remember Size of matrices is Θ(n 2 ), NOT Θ(n) We get the recurrence T (n) 8T (n/2) + Θ(n 2 ). Since log 2 (8) 3, the Master Theorem yields T (n) Θ(n 3 ). (No improvement over MatMult... why? CLSS?...) DS (2014/15) Lecture 3 slide 9 näive divide-and-conquer algorithm (cont d) ssume n is a power of 2. lgorithm D&C-MatMult(, ) 1. n number of rows of 2. if n 1 then return (a 11 b 11 ) 3. else 4. Let ij, ij (for i, j 1, 2) be (n/2 n/2)-submatrices s.th. 11 12 11 12 and 21 22 21 22 5. Recursively compute 11 11, 12 21, 11 12, 12 22, 21 11, 22 21, 21 12, 22 22 6. Compute C 11 11 11 + 12 21, C 12 11 12 + 12 22, C 21 21 11 + 22 21, C 22 21 12 + 22 22 C 11 C 12 7. return Strassen s algorithm (1969) DS (2014/15) Lecture 3 slide 11 ssume n is a power of 2. Let 11 12 11 12 and. 21 22 21 22 We want to compute 11 11 + 12 21 11 12 + 12 22 21 11 + 22 21 21 12 + 22 22 C11 C 12. Strassen s algorithm uses a trick in applying Divide-and-Conquer. DS (2014/15) Lecture 3 slide 10 DS (2014/15) Lecture 3 slide 12
Let Then Strassen s algorithm (cont d) P 1 ( 11 + 22 )( 11 + 22 ) P 2 ( 21 + 22 ) 11 P 3 11 ( 12 22 ) P 4 22 ( 11 + 21 ) P 5 ( 11 + 12 ) 22 P 6 ( 11 + 21 )( 11 + 12 ) P 7 ( 12 22 )( 21 + 22 ) C 11 P 1 + P 4 P 5 + P 7 C 12 P 3 + P 5 C 21 P 2 + P 4 C 22 P 1 + P 3 P 2 + P 6 ( ) ( ) Strassen s algorithm (cont d) Crucial Observation Only 7 multiplications of (n/2 n/2)-matrices are needed to compute. lgorithm Strassen(, ) 1. n number of rows of 2. if n 1 then return (a 11 b 11 ) 3. else 4. Determine ij and ij for i, j 1, 2 (as before) 5. Compute P 1,..., P 7 as in ( ) 6. Compute C 11, C 12, C 21, C 22 as in ( ) C11 C 12 7. return DS (2014/15) Lecture 3 slide 13 Checking Strassen s algorithm - C 11 We will check the equation for C 11 is correct. Strassen s algorithm computes C 11 P1 + P4 P5 + P7. We have P1 (11 + 22)(11 + 22) 1111 + 1122 + 2211 + 2222. P4 22( 11 + 21) 2221 2211. P5 (11 + 12)22 1122 + 1222. P7 (12 22)(21 + 22) 1221 + 1222 2221 2222. Then P1 + P4 1111 + 1122 + 2222 + 2221. Then P1 + P4 P5 1111 + 2222 + 2221 1222. Then P1 + P4 P5 + P7 1111 + 1221, which is C11. homework: check other 3 equations. nalysis of Strassen s algorithm DS (2014/15) Lecture 3 slide 15 Let T (n) be the number of arithmetic operations performed by Strassen. Lines 1 4 and 7 require Θ(1) arithmetic operations Line 5 requires 7T (n/2) + Θ(n 2 ) arithmetic operations Line 6 requires Θ(n 2 ) arithmetic operations. remember. We get the recurrence T (n) 7T (n/2) + Θ(n 2 ). Since log 2 (7) 2.807 > 2, the Master Theorem yields T (n) Θ(n log 2 (7) ). DS (2014/15) Lecture 3 slide 14 DS (2014/15) Lecture 3 slide 16
reakthroughs on matrix multiplication Coppersmith & Winograd (1987) came up with an improved algorithm with running time of... many years of silence... Θ(n 2.376 ). Then in his 2010 PhD thesis, ndrew Stothers from the School of Maths, at the University of Edinburgh got an algorithm with Θ(n c ) for c < 2.3737... Coppersmith/Winograd not optimal. ut Stothers didn t publish. Reading ssignment [CLRS] (3rd ed) Section 4.5 The Master method for solving recurrences (Section 4.3 Using the Master method of [CLRS], 2nd ed) [CLRS] (3rd ed) Section 4.2 (Section 28.2 of [CLRS], 2nd ed) Problems 1. Exercise 4.5-2 of [CLRS] (3rd ed) Exercise 4.3-2 of [CLRS], 2nd ed. 2. Exercise 4.2-1 of [CLRS], 3rd ed. Exercise 28.2-1 [CLRS], 2nd ed. 3. Week 3 tutorial sheet :-) In December 2011, Virginia Vassilevska Williams of Stanford, came up with a Θ(n c ) algoithm, for c < 2.3727 (partly, but not only, making use of some of Stothers ideas) Remarks on Matrix Multiplication DS (2014/15) Lecture 3 slide 17 In practice, the school MatMult algorithm tends to outperform Strassen s algorithm, unless the matrices are huge. The best known lower bound for matrix multiplication is DS (2014/15) Lecture 3 slide 19 Ω(n 2 ). This is a trivial lower bound (need to look at all entries of each matrix). mazingly, Ω(n 2 ) is believed to be the truth Open problem: Can we find a O(n 2+o(1) )-algorithm for Matrix Multiplication of n n matrices? DS (2014/15) Lecture 3 slide 18