Solving recurrences. Frequently showing up when analysing divide&conquer algorithms or, more generally, recursive algorithms.

Transcription

1 Solving recurrences Frequently showing up when analysing divide&conquer algorithms or, more generally, recursive algorithms Example: Merge-Sort(A, p, r) 1: if p < r then 2: q (p + r)/2 3: Merge-Sort(A, p, q) 4: Merge-Sort(A, q + 1, r) 5: Merge(A, p, q, r) 6: end if Initial call for input array A = A[1], A[2],..., A[n] is Merge-Sort(A, 1, n). 1

2 Example: sorted sequence initial sequence Merge-Sort splits length-l sequence into two length-l/2 sequences sorts them recursively merges the two sorted subsequences 2

3 General approach for (simple) D&Q algorithms Let T (n) be running time on problem of size n If n is small enough (say, n c for constant c), then straightforward solution takes Θ(1) If division of problem yields a subproblems, each of which 1/b of original (Merge-Sort: a = b = 2) Division into subproblems takes D(n) Combination of solutions to subproblems takes C(n) Then, T (n) = { Θ(1) if n c at (n/b) + D(n) + C(n) otherwise 3

4 For Merge-Sort: a = b = 2 D(n) = Θ(1) (just compute middle of array) C(n) = Θ(n) (merging has running time linear in length of resulting sequence; take my word for it) Thus T MS (n) = { Θ(1) if n 2 2T (n/2) + Θ(n) otherwise That s what s called a recurrence But: we want closed form!!!!!! 4

5 There are a few methods for solving recurrences, some easy and not powerful, some complicated and powerful. Can t have all of it, eh? 1. guess & verify (also called substitution method ) 2. recursion-tree method 3. master method 4. generating functions plus some others. We re going to see 1, some of 2, and 3. 5

6 Substitution method Basic idea: 1. guess form of solution 2. Use mathematical induction to find constants and show that solution works Believe it or not, in many cases the (by far) harder part is 1. Back to example: we had for n 3. T MS (n) = 2T (n/2) + Θ(n) If you hadn t seen something like this before, how would you guess? 6

7 Practice. Gain some feeling for recurrences. Practice. Gain experience. Practice. One way would be to have a look at a few terms, say if we had T (n) = 2T (n/2) + 3n, then T (n) = 2T (n/2) + 3n = 2(2T (n/4) + 3(n/2)) + 3n = 2(2(2T (n/8) + 3(n/4)) + 3(n/2)) + 3n = 2 3 T (n/2 3 ) (n/2 2 ) (n/2 1 ) (n/2 0 ) We can do this log n times 2 log n T (n/2 log n ) + = n T (1) + 3n = n T (1) + 3n log n log(n) 1 i=0 log(n) 1 i=0 1 2 i 3(n/2 i ) doesn t look too bad for an answer, does it? 7

8 Anyways, that s certainly no proper proof. After guessing a solution you ll have to prove correctness. BTW, the mentioned recursion-tree method is just a slightly more educated version of the plain old guessing. Reading assignment: Section 4.2 Another example. Suppose we have T (n) = 2T ( n/2 ) + n. Guess might be T (n) = O(n log n). We prove T (n) cn log n for appropriate choice of constant c > 0. 8

9 Start: assume that bound holds for n/2, i.e., T ( n/2 ) c n/2 log( n/2 ) and see what happens: T (n) = 2T ( n/2 ) + n 2(c n/2 log( n/2 )) + n cn log(n/2) + n = cn log n cn log 2 + n = cn log n cn + n = cn log n (c 1)n cn log n for c 1 So far so good... looks as though our hypothesis might be true for real values of n. But what about the boundary conditions? It s got to hold everywhere! 9

10 Can t be helped, we ve got to check. By showing that boundary conditions of recurrence are suitable as base cases for the inductive proof. Last example: have got to show that we can choose c large enough s.t. bound T (n) cn log n works for boundary conditions as well. Not always that trivial to show and/or to construct. Assume T (1) = 1 is only such condition, natural enough. For n = 1, T (n) cn log n = c 1 0 = 0 is a bit of a problem, because T (1) 0 and T (1) = 1 don t go together too well simultaneously. 10

11 So, does this mean that we re doomed? Fortunately not (necessarily). Recall that we wanted to prove T (n) = O(n log n). Also, recall that by def of O(), we are free to disregard a constant number of nasty small values of n: f(n) = O(g(n)) constants c, n 0 : f(n) c g(n) for n n 0 A way out of our problem is to remove difficult boundary condition T (1) = 1 from consideration in inductive proof. Note: for n 4, T (n) does not depend directly on T (1) (T (2) = 2T (1) + 2, T (3) = 2T (1) + 3, T (4) = 2T (2) + 4) 11

12 This means: We replace T (1) by T (2) and T (3) as base cases in the inductive proof, letting n 0 = 2. Note: distinguish between base case of recurrence (n = 1) and base cases of inductive proof (n = 2 and n = 3). In other words, we re not cheating, i.e., we re still having the T (1) boundary condition in the recurrence; we re just not taking care of it in the inductive proof. By recurrence T (n) = 2T ( n/2 )+n we get T (2) = 2T (1) + 2 = = 4 and T (3) = 2T (1) + 3 = = 5. Ready to finish the proof that T (n) cn log n for n large enough (i.e., n 2): we re fine whenever T (2) c2 log 2 and T (3) c3 log 3. Any c 2 is OK for both cases (c 1 was OK for rest). Warning: General technique! Can be used very often! 12

13 About guessing solutions No general way, unfortunately. Requires experience (and luck, somet^h^h^h^h^husually). Consider T (n) = 2T ( n/2 + 25) + n Looks similar to last example, but is the additional 25 in the argument going to messy? Not really, because for large n, difference between T ( n/2 ) and T ( n/2 + 25) is not large: both cut n nearly in half: for n = 2, 000 we have T (1, 000) and T (1, 025), for n = 1, 000, 000 we have T (500, 000) and T (500, 025). Thus, reasonable assumption is that now T (n) = O(n log n) as well. 13

14 Also, stepwise refinement sometimes useful. For T (n) = 2T ( n/2 ) + n we see T (n) = Ω(n) (because of the n term) T (n) = O(n 2 ) (easily proven) From there, we can perhaps converge on asymptotically tight bound Θ(n log n). Warning: do not try to compute first, say, x million terms of recurrence and plot the graph. With 32- bit integers, it s hard to tell a logarithmic factor from a log log or even a constant. Constant vigilance: log 2 32 = 32, log log 2 32 = log 32 = 5. There are quite a few constant greater than 5 or 32 around, so hard to tell whether something is c 1 n or c 2 n log log n or c 3 n log n. 14

15 Another not uncommon pitfall: Might be tempted to falsely prove that for T (n) = 2T ( n/2 ) + n we have T (n) = O(n). Guess T (n) c n. T (n) = 2T ( n/2 ) + n 2(c n/2 ) + n cn + n = (c + 1) n = O(n) Although (c + 1) n certainly is O(n), we have not proved that T (n) c n. Must prove exact form of inductive hypothesis! 15

16 A neat trick called changing variables Suppose we have T (n) = 2T ( n ) + log n Now that s some beast, isn t it? Firstly, let s forget about the rounding and consider T (n) = 2T ( n) + log n instead. Now rename m = log n 2 m = n. We know n = n 1/2 = (2 m ) 1/2 = 2 m/2 and thus obtain T (2 m ) = 2T (2 m/2 ) + m Now rename S(m) = T (2 m ) and get S(m) = 2S(m/2) + m Looks familiar? Voila! Know solution S(m) = O(m log m). Going back from S(m) to T (n) we obtain T (n) = T (2 m ) = S(m) = O(m log m) = O(log n log log n) 16

17 The Master Method Recipe for recurrences of the form T (n) = at (n/b) + f(n) with a 1 and b > 1 constant, and f(n) an asymptotically positive function (f(n) = 5, f(n) = c log n, f(n) = n, f(n) = n 12 are just fine). Split problem into a subproblems each of size n/b. Subproblems are solved recursively, each in time T (n/b). Dividing problem and combining solutions of subproblems is captured by f(n). Deals with many frequently seen recurrences (in particular, our Merge-Sort example with a = b = 2 and f(n) = Θ(n)). 17

18 Theorem. Let a 1 and b > 1 bee constants, let f(n) be a function, and let T (n) be defined on the nonnegative integers by the recurrence T (n) = at (n/b) + f(n), where we interpret n/b to mean either n/b or n/b. Then T (n) can be bounded asymptotically as follows. 1. If f(n) = O(n (log b a) ɛ ) for some constant ɛ > 0, then T (n) = Θ(n log b a ). 2. If f(n) = Θ(n log b a ), then T (n) = Θ(n log b a log n). 3. If f(n) = Ω(n (log b a)+ɛ ) for some constant ɛ > 0, and if a f(n/b) c f(n) for some constant c < 1 and all sufficiently large n, then T (n) = Θ(f(n)). 18

19 Note: Although it s looking rather scary, it really isn t. For instance, with Merge-Sort s recurrence T (n) = 2T (n/2) + Θ(n) we have n log b a = n log 2 2 = n 1 = n, and we can apply case 2. The result is therefore Θ(n log b a log n) = Θ(n log n). Another note: f(n) = n (log b a) ɛ = n log b a /n ɛ = o(n log b a ), so the ɛ does matter. This case is basically about small functions f. Yet another note: f(n) = n (log b a)+ɛ = n log b a n ɛ = ω(n log b a ), so the ɛ does matter again. This case is basically about large functions n. The idea is that we compare n log b a to f(n). Result is (intuitively) determined by larger of two. In case 1, n log b a is larger, so result is Θ(n log b a ). In case 3, f(n) is larger, so result is Θ(f(n)). In case 2, both are same size, we multiply by logarithmic factor, and result is Θ(n log b a log n) = Θ(f(n) log n). 19

20 Important: Does not cover all possible cases. For instance, there is a gap between cases 1 and 2 whenever f(n) is smaller than n log b a but not polynomially smaller (homework). 20

21 Using the master theorem Simple enough. Some examples (from the book): T (n) = 9T (n/3) + n We have a = 9, b = 3, f(n) = n. Thus, n log b a = n log 3 9 = n 2. Clearly, f(n) = O(n log 3(9) ɛ ) for ɛ = 1, so case 1 gives T (n) = Θ(n 2 ). T (n) = T (2n/3) + 1 We have a = 1, b = 3/2, and f(n) = 1, so n log b a = n log 2/3 1 = n 0 = 1. Apply case 2 (f(n) = Θ(n log b a ) = Θ(1), result is T (n) = Θ(log n). T (n) = 3T (n/4) + n log n We have a = 3, b = 4, and f(n) = n log n, so n log b a = n log 4 3 = O(n ). Clearly, f(n) = n log(n) = Ω(n) and thus also f(n) = Ω(n log b(a)+ɛ ) for ɛ 0.2. Also, a f(n) = 3(n/4) log(n/4) (3/4)n log n = c f(n) for c = 3/4. Thus we can apply case 3 with result T (n) = Θ(n log n). 21