4.4 Recurrences and Big-O

Transcription

1 4.4. RECURRENCES AND BIG-O Recurrences and Big-O In this section, we will extend our knowledge of recurrences, showing how to solve recurrences with big-o s in them, recurrences in which n b k, and those that do not fit the master theorem. Big-O in Recurrences We have been discussing recurrences where the non-recursive term is something like n or n.however, when talking about algorithms, it is common to have recurrences where the non-recursive term is O(n) oro(n ). We may also have recurrences where the non-recursive term is Θ(n) or Θ(n ) What does at (n/b) + O(g(n)) mean? Is it really an equation? 4.4- Carefully prove by induction that has a solution of the form O(n log n). T (n/) + O(n) if n O(1) if n =1 In order to understand what Exercise means, we have to think about using the definition of big-o, and our interpretation from recursion trees. This interpretation says that in order to solve a problem of size n, we must solve recursively a problems of size n/b and do O(g(n)) additional work. In other words, there exist positive constants c and n 0, such that the amount of work we have to do is at most cg(n), as long as n>n 0. We don t know what the this n 0 is either, and we will just assume, for now, that it is 1. Now, the amount of work for a problem of size n is not cg(n), but at most cg(n), so we rewrite the recurrence as T (n) at (n/b)+cn, for some constant c. We say for some constant c, because we know that there is a constant c that is implicit in the definition of the big-o. We don t know what this constant is, but we know that it exists and is well-defined. We could still solve such a recurrence with a recursion tree, replacing work with an upper bound on work cg(n) on the first level, etc. We can also use the method we used a few lectures ago, guessing the solution and verifying by induction. We use this method for the recurrence in Exercise Here we wish to show that O(n log n). From the definion of big-o, we see that our recurrence has the form T (n/) + cn if n T (n) O(1) if n =1, for some positive real number c. Ussing the definition of big-o again, we can see that we wish to show that T (n) kn log n for some positive constant k. To prove that T (n) kn log n, we start with a base case. If we choose n = 1, we get T (1) k1log1 or T (1) 0. But we said that T (1) = O(1) so this is actually false. Fortunately

2 9 CHAPTER 4. INDUCTION, RECURSION AND RECURRENCES we are ultimately trying to prove a big-o statement about T (n) so our bound doesn t have to hold for all n, justforn>n 0. So choosing n 0 = 1, we have a base case of. When n = the right side of our desired inequality T (n) kn log n becomes kn log n =klog = k. Comparing this with T(), we see that as long as we choose k>t()/, our base case will hold. For the general case, we assume that for n <n, T (n ) kn log n. Now we will bound T (n) using the inductive hypothesis in the second inequality below: T (n) T (n/) + cn k(n/) log(n/) + cn kn log n kn log + cn = kn log n +(c k)n. So our proof will work as long as kn log n +(c k)n <knlog n which is equivalent to (c k)n 0 or k c. But this is fine, because we get to pick the k, so we will pick it to be any value larger than c. We will also pick it so that k>t()/, because of the base case. In other words, we know that O(n log n) because, for whatever constant is implicit in the O(g(n)), we can choose a larger constant to make the statement O(n log n) true. In this case, if we choose k =maxt ()/,c, we will have chosen correctly. This order of picking constants occurs in all these types of proofs. This order holds because we are given O(g(n)) in the recurrence, therefore we know that there exists a constant c such that the O(g(n)) is really at most cg(n). This is true regardless of what recurrence we have for T. We are trying to prove a big-o statement about T (n), so in this case we must show that there exists a constant k with T (n) k(something); because c is independent of T we can define k in terms of c. Recurrences for general n So far, for a recurrence of the form at (n/b)+g(n), we have assumed that n is an integral power of b. This is very convenient because it means that every subproblem that we see is of integral size. Of course, not all problems are of a nice size like this, and we would like a method that works for all n. If you are willing to drop the analogy between recursion trees and recursive algorithms, and just think about recursion trees, you see that we never actually required that the subproblems be of integral size. We can also extend the recursion tree method to deal with the number of problems being non-integral too. Again we have to think a bit more abstractly and all the possibility of fractional sized subproblems. We will not pursue the details here. In fact there is another simple solution to the problem that n may not be a power of b. Since we are only concerned with the solution to T (n) in big-o terms, that is, up to constant factors, the value of T (n) and the value of T (bn) are often not too far apart. In the lemma that follows, we capture the idea of T (n) andt (bn) not being too far apart by the condition that T (bn) =O(T (n). When we say T (x) =O(1) if 0 x<b,wearesayingthatt (x) is bounded by a constant on the whole interval. This is slightly different from the way we have used big Oh notation before, but is quite common. Lemma Let a and b be positive real numbers with b>1 and let f : R + R + be a function such that f(bx) =O(f(x)). Then for any solution t(x) to the recurrence at(x/b)+f(x) if x b t(x) O(1) if 0 x<b,

3 4.4. RECURRENCES AND BIG-O 93 t(x) =O(h(x)) if for every solution T(x) to the recurrence we have T (x) =O(h(x)). T (b n ) at (b n /b)+f(b n ) if n 0 O(1) if n =0, To prove the lemma we would simply write down the result of iterating both recurrences and compare them; a recursion tree would also illustrate this process. Notice that this lemma relies on a condition on f. We note that this condition holds for all polynomially bounded functions. We have been sloppy in our use of recurrences in that we have not paid attention to floors and ceilings. If we allow n to not be a power of b this becomes an issue. We can see that by taking floors, we can never cause a problem, as values only get smaller with floors. We can also show that, even if we take ceilings (as any algorithm that does provide a perfectly even split must do), our recurrences still have the same solutions. Lemma 4.4. Let a and b be positive real numbers with b and let f : R + R + be a function such that f(bx) =O(f(x)). Then for any solution t(x) to the recurrence t(x) at( x/b )+f( x ) if x b O(1) if 0 x<b, t(x) =O(h(x)) if for every solution T(x) to the recurrence we have T (x) =O(h(x)). T (x) at (x/b)+f(x) if x b O(1) if 0 x<b, The proof of this lemma is essentially the same as the proof of the previous one. One can actually show that this is true even with b 1, see CLR for details. Further wrinkles Show by induction that for any solution T (n) to the following recurrence, O(n): T (n/3) + O(n) if n 3 O(1) otherwise Show by induction that for any solution T (n) to the following recurrence, O(n ): 4T (n/) + O(n) if n O(1) otherwise

4 94 CHAPTER 4. INDUCTION, RECURSION AND RECURRENCES Implicit in the recursive statement of Exercise that T (n/3)+o(n) is the existence of two constants n 0 and c such that T (n) T (n/3) + cn if n>n 0. For the sake of your intuition, imagine that these ocnstants are in sealed envelopes that we may not open until after we have solved the problem. Since we know they are in the envelopes, we may use the symbols that stand for them in any way we want. In particular, since we want to show that O(n), we want to find two more constants n 0 and k such that T (n) kn whenever n>n 0.Sinceweknowthat the constants c and n 0 exist (they are in the envelope, after all), we can use them if we need to in describing k and n 0. There are two ways to finish Exercise One uses Lemma and one does not. We will outline both ways so you can see the diffference. We begin with the version that does not use the Lemma. Since the exercise did not tell us to assume n is a power of 3, we observe that we can reach many different numbers in the range 1 x<3 by dividing larger integers by higher and higher powers of 3. Thus for our base case we have to consider all numbers x with 1 x<3. The base case of the recurrence tells us that there is some constant q such that T (x) q for 1 x<3. Thus T (x) qx for 1 x<3 because x 1. Thus one possible value for k is q. Now we don t know what k is yet, so it is rather frustrating to be trying to prove something about it. Instead, let us experiment with assuming someone else has witten down a good k in yet another envelope and see if we can figure out anything about what they wrote, or whether it doesn t matter what they wrote and k = q will work just fine. Thus we assume we know that T (y) <kywhen 3 y x/3. Then by this assumption and our recurrence, T (x) k(x/3) + cx/3 =kx kx/3+cx = kx +(c k/3)x. (4.18) We can see from this that if we choose k =3c/ or any bigger number, then an inductive proof should work out because then the coefficient of x will be 0 or negative.. But how did we know that we should write the equalities we did in Equation 4.18? Since we were trying to get T (x) kx, it made sense to write kx kx/3 inplaceofkx/3, and combining the c with the k/3 wasa rather natural piece of arithmetic. However this is not really our proof. We will now prove by induction that T (n/3) + O(n) if n 3 O(1) otherwise. Since dividing an integer value by 3 can give a non-integer value, we prove the recurrence for values x that are an integer divided by a power of 3. We are given that there is a constant q>0 such that T (x) q if 1 x<3. We also know that there is an n 0 such that T (x) T (x/3) + cx for x>n 0. Further, for a given x with 3 x n 0, there are only a finite number of values of x/3 i that are larger than or equal to 3. Thus, given x we let k be the largest of q, 3c/ andthe values T (x/3 i )for3 x/3 i n 0. Then by the choice of k, T (y) ky when 1 y<n 0.Nowif x>n 0 assume inductively that when y = x/3 wehavet (y) ky. Then T (x) T (x/3) + cx k(x/3) + cx = kx +(c k/3)x kx, since (c k/3) is negative or zero because k 3c/. Therefore by the principle of mathematical induction, T (x) kx for all real numbers x 1, so that T (x) =O(x). Here is a new way that we are using mathematical induction. Because for any real number x the sequence of values x/3 i that are greater than 1 is finite, we can apply mathematical induction to reason about the terms

5 4.4. RECURRENCES AND BIG-O 95 of this sequence. (Notice that we never had to specify n 0. Thus we are taking n 0 to be 1. We could have simplified the choice of k somewhat had we taken n 0 to be n 0. Then we could have chosen for k the larger of 3c/ andt (x/3 i ) for the smallest i such that x/3 i n 0. Ignore this parenthetical comment if it confuses you!) If we decide to use Lemma 4.4.1, we can procede in Exercise as before. We are given the constants q, n 0 and c already. We are only going to prove our recurrence for values of n that arepowersof3. Wewishtoprovetherearepositiveconstantsn 0 and k such that T (n) kn for all n>n 0. We will choose n 0 = 1 here. As above, if we assume T (n) kn, wecanwrite T (n) T (n/3) + cn k(n/3) + cn = kn +(c k 3 )n In this case, as long as c k 3 0, the inductive step proof will be correct. Now we choose k to be the largest of the maximum of T (n) for1 n n 0, q, and3c/. To prove by induction that T (x) kx we begin by observing that when 1 n n 0 we have T (n) T (n) 1 kn. Next we assume inductively that n>n 0 and for m with 1 m<nwe have T (m) km. Nowwemaywrite T (n) T (n/3) + cn kn/3+cn = kn +(c k/3)n kn, becausewechosek to be at least as large as 3c/, making c k/3 negative or zero. Thus by the principle of mathematical induction we have T (n) kn for all n 1andsoO(n). Now let s analyze Exercise We won t dot all the i s and cross all the t s here because there is only one major difference between this exercise and the previous one. Let c be the constant implicit in the O(n). We wish to prove there are a n 0 and a k that T (n) kn for n>n 0. Assuming that the base case holds, we can bound T (n) inductively as follows: ( ) n T (n) 4T + cn ( ) n 4(k )+cn ( ) kn = 4 + cn 4 = kn + cn. To procede as before, we would like to choose a value of k so that cn < 0. But we see that we have a problem because both c and n are always positive! What went wrong? We have a statement that we know is true, and we have a proof method (induction) that worked nicely for similar problems. The problem is that, while the statment is true, it is too weak to be proved by induction. Let s try to prove something that is actually stronger, namely that for some positive constants k 1 and k, T (n) k 1 n k n. Now proceding as before, we get T (n) 4T (n/) + cn

6 96 CHAPTER 4. INDUCTION, RECURSION AND RECURRENCES ( ) n ( ) n 4(k 1 k )+cn ( k1 n ( ) ) n = 4 k + cn 4 = k 1 n k n + cn = k 1 n k n +(c k )n. Nowwehavetomake(c k )n>0 for the last line to be at most k 1 n k n, and so we just choose k <c.sincet (n) k 1 n + k n for some constants k 1 and k,thent(n) =O(n ). At first glance, this seems like a paradox: why is it easier to prove a stronger statement than it is to prove a weaker one. This phenomenom happens often in mathematics: a stronger statement is often easier to prove than a weaker one. It happens particularly often when using induction. Think carefully about an inductive proof where you have assumed that a bound holds forvaluessmallerthann and you are trying to prove a statement for n. You use the bound you have proved for smaller values to help prove the bound for n Thus if the bound you used for smaller values is actually weak, then that is hindering you in proving the bound for n. Inother words when you want to prove something about p(n) you are using p(1)... p(n 1). Thus if these are stronger, they will be of greater help in proving p(n). In the case above, the problem was that these values, p(1)...p(n 1) were too weak, and thus we were not able to prove p(n). By using a stronger p(1)...p(n 1), however, we were able to prove p(n). Functions other than n c It is also common to see recurrences in with the additional term is not of the form n c, but of some other form. The most common such is example is when that term has logarithms. For example, consider the recurrence: T (n/) + n log n if n>1. O(1) n = 1 Just as before, we can draw a recursiont tree; the whole methodolgy works, but our sums may be a little more complicated. The tree for this recurrence is as follows. Problem Size Work n n log n levels n/ n/4 n/ log(n/) + n/ log(n/) = n log(n/) 4(n/4 log(n/4)) = n log(n/4) n/8 8(n/8 log(n/8)) = n log(n/8) 1 n( n/n log(n/n)) = n

7 4.4. RECURRENCES AND BIG-O 97 This is similar to the tree for T (n/) + n, except that the work for a problem of size i is ilogi. Thus if we sum the work and evaluate the sum we get log n ( ) n n log i log n ( ) n = n log i log n = n (log n log i ) log n log n = n( log n log i ) log n = n(log n(log n +1)+ i) = n(log n(log n +1)+log n(log n +1)/) = O(n log n) As another example consider the recurrence T (n/) + n log n if n>1 O(1) if n =1. The tree for this recurrence is Problem Size Work n n log n levels n/ n/4 n/ log(n/) n/4 log(n4) n/8 n/8 log(n/8) 1 1 Summing the work, we get log n ( ) n n i log i log n 1 = n i (log n log i ) = n(log n log n n(log n()) = O(n log n). ( 1 ) i ( i i )

8 98 CHAPTER 4. INDUCTION, RECURSION AND RECURRENCES Exercises E4.4-1 Find the best big-oh upper bound you can to any solution to the recurrence T ( n/ +1)+O(n) if n 4 O(1) if n<4. E4.4- Find the best big-oh upper bound you can to any solution to the recurrence 4T (n/) + n log n if n>1 O(1) if n =1. E4.4-3 Give an example of a function for which f(bx) =O(f(x)). Give an example of a function for which f(bx) isnoto(f(x)). E4.4-4 Show by induction that has O(n 3 ) for any solution T (n). 8T (n/) + n log n if n>1 O(1) if n =1 E4.4-5 Show by induction that any solution to a recurrence of the form T (n/3) + O(log3 n) if n 3 O(1) if 1 n<3 is O(n log 3 n). What happens if you replace by 3 (explain why)? Would it make a difference if we used a different base for the logarithm (only an intuitive explanation is needed here). E4.4-6 What happens if you replace the in the previous exercise by 4? replace the log 3 by log b? (Use recursion trees.) What if you