Introduction to Algorithms 6.046J/18.401J/SMA PDF Free Download

Introduction to Algorithms 6.046J/8.40J/SMA5503 Lecture 3 Prof. Piotr Indyk

The divide-and-conquer design paradigm. Divide the problem (instance) into subproblems. 2. Conquer the subproblems by solving them recursively. 3. Combine subproblem solutions. February, 2003 (c) Charles Leiserson and Piotr Indyk L3.2

Example: merge sort. Divide: Trivial. 2. Conquer: Recursively sort 2 subarrays. 3. Combine: Linear-time merge. T(n) = 2 T(n/2) + O(n) # subproblems subproblem size work dividing and combining February, 2003 (c) Charles Leiserson and Piotr Indyk L3.3

Binary search Find an element in a sorted array:. Divide: Check middle element. 2. Conquer: Recursively search subarray. 3. Combine: Trivial. Example: Find 9 3 5 7 8 9 2 5 February, 2003 (c) Charles Leiserson and Piotr Indyk L3.4

Recurrence for binary search T(n) = T(n/2) + Θ() # subproblems subproblem size work dividing and combining n log ba = n log 2 = n 0 = CASE 2 (k = 0) T(n) = Θ(lg n). February, 2003 (c) Charles Leiserson and Piotr Indyk L3.0

Powering a number Problem: Compute a n, where n N. Naive algorithm: Θ(n). Divide-and-conquer algorithm: a n = a n/2 a n/2 a (n )/2 a (n )/2 a if n is even; if n is odd. T(n) = T(n/2) + Θ() T(n) = Θ(lg n). February, 2003 (c) Charles Leiserson and Piotr Indyk L3.

Polynomial multiplication Input: a(x)=a 0 +a x+ +a n x n, b(x)=b 0 +b x+ +b n x n, Output: c(x)= a(x)*b(x) =c 0 +c x+ +c 2n x 2n c i =a 0 b i +a b i- + + a i- b +a i b 0 Example: (a 0 +a x) * (b 0 +b x) = a 0 b 0 + (a 0 b +a b 0 )x + a b x 2 = c 0 + c x + c 2 x 2 February, 2003 (c) Charles Leiserson and Piotr Indyk L3.2

Motivation (more in recitations) Essentially equivalent to multiplying large integers: 6046*600 = (6*0 0 + 4*0 + 0*0 2 + 6*0 3 ) * (*0 0 + 0 * 0 + 0*0 2 + 6*0 3 ) = a(0) * b(0) = c(0), where c(x)=a(x)*b(x) c(0) = c 0 0 0 + c 0 + + c 6 0 6 The coefficients of c form the digits of the product c(0) February, 2003 (c) Charles Leiserson and Piotr Indyk L3.3

How to multiply two polynomials From the definition: Θ(n 2 ) time Faster? Use divide and conquer Divide: a(x) = a 0 +a x+ +a n x n = (a 0 + + a n/2 x n/2 ) + x n/2 (a n/2 x 0 + + a n x n/2 ) = p(x) + x n/2 q(x) = p + x n/2 q In the same way: b(x) = s+x n/2 t February, 2003 (c) Charles Leiserson and Piotr Indyk L3.4

Conquer Observe that: a*b = (p+x n/2 q) * (s+x n/2 t) = p*s + x n/2 (p*t+q*s) + x n q*t But p,q,s,t have degree n/2 can compute the products recursively! (and then perform Θ(n) additions) February, 2003 (c) Charles Leiserson and Piotr Indyk L3.5

The great moment T(n) = 4 T(n/2) + Θ(n) # subproblems subproblem size work dividing and combining n log ba = n log 2 4 = n 2 CASE T(n) = Θ(n 2 ). No better than the ordinary algorithm??? February, 2003 (c) Charles Leiserson and Piotr Indyk L3.6

Need to be more clever Compute: p*s q*t (p+q) * (s+t) = p*s + (p*t + q*s) + q*t (all polynomials have degree n/2 ) Can extract (p*t + q*s) without any additional multiplications! February, 2003 (c) Charles Leiserson and Piotr Indyk L3.7

The truly great moment T(n) = 3T(n/2) + Θ(n) # subproblems subproblem size work adding and subtracting n log ba = n log 2 3 = n.58496 Much better than Θ(n 2 )! February, 2003 (c) Charles Leiserson and Piotr Indyk L3.8

Matrix multiplication Input: A = [a ij ], B = [b ij ]. Output: C = [c ij ] = A B. i, j =, 2,, n. c c 2 c c 2 22 c c n 2n = a a 2 a a 2 22 a a n 2n b b 2 b b 2 22 b b n 2n c n c n2 c nn a n a n2 a nn b n b n2 b nn c ij = n k= a ik b kj February, 2003 (c) Charles Leiserson and Piotr Indyk L3.9

Standard algorithm for i to n do for j to n do c ij 0 for k to n do c ij c ij + a ik b kj Running time = Θ(n 3 ) February, 2003 (c) Charles Leiserson and Piotr Indyk L3.20

Divide-and-conquer algorithm IDEA: n n matrix = 2 2 matrix of (n/2) (n/2) submatrices: r t s u = a c b d e g f h r = ae + bg s = af + bh t = ce + dg u = cf + dh C = A B 8 mults of (n/2) (n/2) submatrices 4 adds of (n/2) (n/2) submatrices February, 2003 (c) Charles Leiserson and Piotr Indyk L3.2

Analysis of D&C algorithm T(n) = 8 T(n/2) + Θ(n 2 ) # submatrices submatrix size work adding submatrices n log ba = n log 2 8 = n 3 CASE T(n) = Θ(n 3 ). No better than the ordinary algorithm. February, 2003 (c) Charles Leiserson and Piotr Indyk L3.22

Strassen s idea (969) Multiply 2 2 matrices with only 7 recursive mults. P = a ( f h) P 2 = (a + b) h P 3 = (c + d) e P 4 = d (g e) P 5 = (a + d) (e + h) P 6 = (b d) (g + h) P 7 = (a c) (e + f ) r = P 5 + P 4 P 2 + P 6 s = P + P 2 t = P 3 + P 4 u = P 5 + P P 3 P 7 7 mults, 8 adds/subs. Note: No reliance on commutativity of mult! February, 2003 (c) Charles Leiserson and Piotr Indyk L3.23

Strassen s idea Multiply 2 2 matrices with only 7 recursive mults. P = a ( f h) P 2 = (a + b) h P 3 = (c + d) e P 4 = d (g e) P 5 = (a + d) (e + h) P 6 = (b d) (g + h) P 7 = (a c) (e + f ) r = P 5 + P 4 P 2 + P 6 = (a + d) (e + h) + d (g e) (a + b) h + (b d) (g + h) = ae + ah + de + dh + dg de ah bh + bg + bh dg dh = ae + bg February, 2003 (c) Charles Leiserson and Piotr Indyk L3.24

Strassen s algorithm. Divide: Partition A and B into (n/2) (n/2) submatrices. Form terms to be multiplied using + and. 2. Conquer: Perform 7 multiplications of (n/2) (n/2) submatrices recursively. 3. Combine: Form C using + and on (n/2) (n/2) submatrices. T(n) = 7 T(n/2) + Θ(n 2 ) February, 2003 (c) Charles Leiserson and Piotr Indyk L3.25

Analysis of Strassen T(n) = 7 T(n/2) + Θ(n 2 ) n log ba = n log 2 7 n 2.8 CASE T(n) = Θ(n lg 7 ). The number 2.8 may not seem much smaller than 3, but because the difference is in the exponent, the impact on running time is significant. In fact, Strassen s algorithm beats the ordinary algorithm on today s machines for n 30 or so. Best to date (of theoretical interest only): Θ(n 2.376 ). February, 2003 (c) Charles Leiserson and Piotr Indyk L3.26

Conclusion Divide and conquer is just one of several powerful techniques for algorithm design. Divide-and-conquer algorithms can be analyzed using recurrences and the master method (so practice this math). Can lead to more efficient algorithms February, 2003 (c) Charles Leiserson and Piotr Indyk L3.27

VLSI layout Problem: Embed a complete binary tree with n leaves in a grid using minimal area. W(n) H(n) H(n) = H(n/2) + Θ() W(n) = 2W(n/2) + Θ() = Θ(lg n) = Θ(n) Area = Θ(n lg n) February, 2003 (c) Charles Leiserson and Piotr Indyk L3.28

H-tree embedding L(n) L(n) L(n) = 2L(n/4) + Θ() = Θ( n) Area = Θ(n) L(n/4) Θ() L(n/4) February, 2003 (c) Charles Leiserson and Piotr Indyk L3.29

Master theorem (reprise) T(n) = a T(n/b) + f (n) CASE : f(n) = O(n log ba ε ) T(n) = Θ(n log ba ). CASE 2: f (n) = Θ(n log ba lg k n) T(n) = Θ(n log ba lg k+ n). CASE 3: f (n) = Ω(n log ba + ε ) and a f (n/b) c f (n) T(n) = Θ( f (n)). Merge sort: a = 2, b = 2 n log ba = n CASE 2 (k = 0) T(n) = Θ(n lg n). February, 2003 (c) Charles Leiserson and Piotr Indyk L3.30

Fibonacci numbers Recursive definition: 0 if n = 0; F n = if n = ; F n + F n 2 if n 2. 0 2 3 5 8 3 2 34 Naive recursive algorithm: Ω(φ n ) (exponential time), where φ = ( + is the golden ratio. 5)/2 February, 2003 (c) Charles Leiserson and Piotr Indyk L3.3

Computing Fibonacci numbers Naive recursive squaring: F n = φ n / 5 rounded to the nearest integer. Recursive squaring: Θ(lg n) time. This method is unreliable, since floating-point arithmetic is prone to round-off errors. Bottom-up: Compute F 0, F, F 2,, F n in order, forming each number by summing the two previous. Running time: Θ(n). February, 2003 (c) Charles Leiserson and Piotr Indyk L3.32

February, 2003 (c) Charles Leiserson and Piotr Indyk L3.33 Recursive squaring n F F F F n n n n = + 0 Theorem:. Proof of theorem. (Induction on n.) Base (n = ):. 0 0 2 = F F F F Algorithm: Recursive squaring. Time = Θ(lg n).

February, 2003 (c) Charles Leiserson and Piotr Indyk L3.34 Recursive squaring.. Inductive step (n 2): n n F F F F F F F F n n n n n n n n = = = + 0 0 0 0 2

Introduction to Algorithms 6.046J/18.401J/SMA5503