Data Structures and Algorithms CMPSC 465

Data Structures and Algorithms CMPSC 465 LECTURE 9 Solving recurrences Substitution method Adam Smith S. Raskhodnikova and A. Smith; based on slides by E. Demaine and C. Leiserson

Review question Draw the recursion tree for the following recurrence: T(n) = 4T(n/2) + c What is the depth? How many leaves? Guess T(n)?

Methods for solving recurrences Recursion tree Examples in last two lectures Substitution method This lecture Master theorem

Substitution method The most general method: 1.Guess the form of the solution. 2.Verify by induction. 3.Solve for constants.

Substitution method The most general method: 1.Guess the form of the solution. 2.Verify by induction. 3.Solve for constants. EXAMPLE: T(n) = 4T(n/2) + n [Assume that T(1) = Θ(1).] Guess O(n 3 ). (Prove O and Ω separately.) Assume that T(k) ck 3 for k < n. Prove T(n) cn 3 by induction.

Example of substitution T ( n) = 4T ( n / 2) + n 4c( n / 2) 3 + n = ( c / 2) n3 + n = cn3 (( c / 2) n3 n) desired residual cn3 desired whenever (c/2)n 3 n 0, for example, if c 2 and n 1. residual

Example (continued) We must also handle the initial conditions, that is, ground the induction with base cases. Base: T(n) = Θ(1) for all n < n 0, where n 0 is a suitable constant. For 1 n < n 0, we have Θ(1) cn 3, if we pick c big enough.

Example (continued) We must also handle the initial conditions, that is, ground the induction with base cases. Base: T(n) = Θ(1) for all n < n 0, where n 0 is a suitable constant. For 1 n < n 0, we have Θ(1) cn 3, if we pick c big enough. This bound is not tight!

A tighter upper bound? We shall prove that T(n) = O(n 2 ).

A tighter upper bound? We shall prove that T(n) = O(n 2 ). Assume that T(k) ck 2 for k < n: T ( n) = 4T ( n / 2) + n 2 4c( n / 2) + n 2 = cn + n 2 = O( n )

A tighter upper bound? We shall prove that T(n) = O(n 2 ). Assume that T(k) ck 2 for k < n: T ( n) = 4T ( n / 2) + n 2 4c( n / 2) + n 2 = cn + n 2 = O( n ) Wrong! We must prove the inductive hypothesis

A tighter upper bound? We shall prove that T(n) = O(n 2 ). Assume that T(k) ck 2 for k < n: T ( n) = 4T ( n / 2) + n 2 4c( n / 2) + n 2 = cn + n 2 = O( n ) Wrong! We must prove the I.H. = cn2 ( n) [ desired residual ] cn2 for no choice of c > 0. Lose!

A tighter upper bound! IDEA: Strengthen the inductive hypothesis. Subtract a low-order term. Inductive hypothesis: T(k) c 1 k 2 c 2 k for k < n.

A tighter upper bound! IDEA: Strengthen the inductive hypothesis. Subtract a low-order term. Inductive hypothesis: T(k) c 1 k 2 c 2 k for k < n. T(n) = 4T(n/2) + n 4(c 1 (n/2) 2 c 2 (n/2)) + n = c 1 n 2 2c 2 n + n = c 1 n 2 c 2 n (c 2 n n) c 1 n 2 c 2 n if c 2 1.

A tighter upper bound! IDEA: Strengthen the inductive hypothesis. Subtract a low-order term. Inductive hypothesis: T(k) c 1 k 2 c 2 k for k < n. T(n) = 4T(n/2) + n 4(c 1 (n/2) 2 c 2 (n/2)) + n = c 1 n 2 2c 2 n + n = c 1 n 2 c 2 n (c 2 n n) c 1 n 2 c 2 n if c 2 1. Pick c 1 big enough to handle the initial conditions.

A matching lower bound IDEA: Formulate inductive hypothesis with Inductive hypothesis: T(k) ck 2 for all k < n. T(n) = 4T(n/2) + n 4 cn 2 /4 + n = cn 2 + n cn 2 for any c 0. Pick c small enough to handle the initial conditions.

Appendix: geometric series 1 1 1 2 x x x = + + + for x < 1 1 1 1 1 2 x x x x x n n = + + + + + for x 1

Data Structures and Algorithms CMPSC 465 LECTURE 10? Solving recurrences Master theorem Adam Smith S. Raskhodnikova and A. Smith; based on slides by E. Demaine and C. Leiserson

Review questions Guess the solution to the recurrence: T(n)=2T(n/3)+n 3/2. (Answer: Θ(n 3/2 ).) Draw the recursion tree for this recurrence. a. What is its height? b. What is the number of leaves in the tree? (Answer: h=log 3 n.) (Answer: n (1/log 3).)

The master method The master method applies to recurrences of the form T(n) = a T(n/b) + f (n), where a 1, b > 1, and f is asymptotically positive, that is f (n) >0 for all n > n 0.

Three common cases Compare f (n) with n log ba : 1. f (n) = O(n log ba ε ) for some constant ε > 0. f (n) grows polynomially slower than n log ba (by an n ε factor). Solution: T(n) = Θ(n log ba ).

Three common cases Compare f (n) with n log ba : 1. f (n) = O(n log ba ε ) for some constant ε > 0. f (n) grows polynomially slower than n log ba (by an n ε factor). Solution: T(n) = Θ(n log ba ). 2. f (n) = Θ(n log ba lg k n) for some constant k 0. f (n) and n log ba grow at similar rates. Solution: T(n) = Θ(n log ba lg k+1 n).

Three common cases (cont.) Compare f (n) with n log ba : 3. f (n) = Ω(n log ba + ε ) for some constant ε > 0. f (n) grows polynomially faster than n log ba (by an n ε factor), and f (n) satisfies the regularity condition that a f (n/b) c f (n) for some constant c < 1. Solution: T(n) = Θ( f (n) ).

Idea of master theorem Recursion tree: f (n) a f (n/b) f (n/b) a f (n/b 2 ) f (n/b 2 ) f (n/b 2 ) f (n/b) Τ (1)

Idea of master theorem Recursion tree: f (n) a f (n/b) f (n/b) a f (n/b 2 ) f (n/b 2 ) f (n/b 2 ) f (n/b) f (n) a f (n/b) a 2 f (n/b 2 ) Τ (1)

Idea of master theorem h = log b n Recursion tree: f (n) a f (n/b) f (n/b) a f (n/b 2 ) f (n/b 2 ) f (n/b 2 ) f (n/b) f (n) a f (n/b) a 2 f (n/b 2 ) Τ (1)

Idea of master theorem h = log b n Recursion tree: f (n) a f (n/b) f (n/b) a f (n/b 2 ) f (n/b 2 ) f (n/b 2 ) f (n/b) f (n) a f (n/b) a 2 f (n/b 2 ) Τ (1) #leaves = a h = a log bn = n log ba n log ba Τ (1)

Idea of master theorem h = log b n Recursion tree: f (n) a f (n/b) f (n/b) a f (n/b 2 ) f (n/b 2 ) f (n/b 2 ) f (n/b) f (n) a f (n/b) a 2 f (n/b 2 ) Τ (1) CASE 1: The weight increases geometrically from the root to the leaves. The leaves hold a constant fraction of the total weight. n log ba Τ (1) Θ(n log ba )

Idea of master theorem h = log b n Recursion tree: f (n) a f (n/b) f (n/b) a f (n/b 2 ) f (n/b 2 ) f (n/b 2 ) f (n/b) f (n) a f (n/b) a 2 f (n/b 2 ) Τ (1) CASE 2: (k = 0) The weight is approximately the same on each of the log b n levels. n log ba Τ (1) Θ(n log ba lg n)

Idea of master theorem h = log b n Recursion tree: f (n) a f (n/b) f (n/b) a f (n/b 2 ) f (n/b 2 ) f (n/b 2 ) f (n/b) f (n) a f (n/b) a 2 f (n/b 2 ) Τ (1) CASE 3: The weight decreases geometrically from the root to the leaves. The root holds a constant fraction of the total weight. n log ba Τ (1) Θ( f (n))

Examples EX. T(n) = 4T(n/2) + n a = 4, b = 2 n log ba = n 2 ; f (n) = n. CASE 1: f (n) = O(n 2 ε ) for ε = 1. T(n) = Θ(n 2 ).

Examples EX. T(n) = 4T(n/2) + n a = 4, b = 2 n log ba = n 2 ; f (n) = n. CASE 1: f (n) = O(n 2 ε ) for ε = 1. T(n) = Θ(n 2 ). EX. T(n) = 4T(n/2) + n 2 a = 4, b = 2 n log ba = n 2 ; f (n) = n 2. CASE 2: f (n) = Θ(n 2 lg 0 n), that is, k = 0. T(n) = Θ(n 2 lg n).

Examples EX. T(n) = 4T(n/2) + n 3 a = 4, b = 2 n log ba = n 2 ; f (n) = n 3. CASE 3: f (n) = Ω(n 2 + ε ) for ε = 1 and 4(n/2) 3 cn 3 (reg. cond.) for c = 1/2. T(n) = Θ(n 3 ).

Examples EX. T(n) = 4T(n/2) + n 3 a = 4, b = 2 n log ba = n 2 ; f (n) = n 3. CASE 3: f (n) = Ω(n 2 + ε ) for ε = 1 and 4(n/2) 3 cn 3 (reg. cond.) for c = 1/2. T(n) = Θ(n 3 ). EX. T(n) = 4T(n/2) + n 2 /lg n a = 4, b = 2 n log ba = n 2 ; f (n) = n 2 /lg n. Master method does not apply. In particular, for every constant ε > 0, we have n ε = ω(lg n).