4.3 Growth Rates of Solutions to Recurrences

Transcription

1 4.3. GROWTH RATES OF SOLUTIONS TO RECURRENCES Growth Rates of Solutios to Recurreces Divide ad Coquer Algorithms Oe of the most basic ad powerful algorithmic techiques is divide ad coquer. You have already see this techique i several places. Cosider, for example, the biary search algorithm, which we will describe i the cotext of guessig a umber betwee 1 ad 100. Suppose someoe asks you to guess a umber betwee 1 ad 100 ad, after each guess, they will tell you whether your guess is correct, too high or too low. The atural way for you proceed is to start by guessig 50. Why is this? Well, either you will guess correctly (if the umber happes to be 50), or you will be told too high or too low. I the former case, you ow kow that the solutio is betwee 1 ad 49, while i the latter you kow that the solutio is betwee 51 ad 100. I either case you ow have to guess from a rage that is oly half as big. Thus you have divided the problem up ito a problem that is oly half as big, ad you ca ow (recursively) coquer this problem. Lettig T () be umber of guesses i the worst case for biary search o items, ad assumig that is a power of 2, we get a recurrece of the followig form: T () = T (/2) + 1 if 2 1 if =1 (4.13) That is, the umber of guesses to carry out biary search o items is equal to 1 step (the guess) plus the time to solve biary search o the remaiig /2 items. What we are really iterested i is how much time it takes to use biary search i a computer program that looks for a item i a ordered list. While the umber of guesses gives us a feel for the amout of time, processig each guess may take several steps i our computer program. The exact amout of time these steps take might deped o some factors we have little cotrol over, such as where portios of the list are stored. Also, we may have to deal with lists whose legth is ot a power of two. Thus a more realistic descriptio of the time eeded would be T () T ( /2 )+C1 if 2 C 2 if =1, (4.14) where C 1 ad C 2 are costats. It turs out that the solutio to (4.13) ad (4.14) are roughly the same (ad expressed i big-o otatio are exactly the same). This is almost always the case; we will come back to this issue. For ow, let us ot worry about floors ad ceiligs ad the distictio betwee thigs that take 1 uit of time ad o more tha some costat amout of time. Let s tur to aother example of a divide ad coquer algorithm, mergesort. I this algorithm, you wish to sort a list of items. Let us assume that the data is stored i a array A i positios 1 through. Mergesort ca be stated as follows: MergeSort(A,low,high) if (low == high) retur else mid = (low + high) /2

2 82 CHAPTER 4. INDUCTION, RECURSION AND RECURRENCES MergeSort(A,low,mid) MergeSort(A,mid+1,high) Merge the sorted lists from the previous two steps You have probably see this algorithm already ad if ot, you ca look i CLR for a good descriptio. Suffice to say that the base case (low = high) takes oe step, while the other case executes 1 step, makes two recursive calls o problems of size /2, ad the executes the Merge step, which ca be doe i steps. Thus we obtai the followig recurrece for the worst-case ruig time of mergesort: T () = 2T (/2) + if >1 1 if =1 (4.15) Recurreces such as this oe ca be uderstood via a recursio tree. This techique, which we itroduce below, will allow us to aalyze recurreces that arise i divide-ad-coquer algorithms, ad those that arise i other recursive algorithms, such as the Towers of Haoi, as well. Recursio Trees We will itroduce the idea of a recursio tree via several examples. It is helpful to have a algorithmic iterpretatio of a recurrece. For example, (igorig for a momet the base case) we ca iterpret the recurrece T () =2T (/2) + (4.16) as i order to solve a problem of size we must solve 2 problems of size /2 ad do uits of additioal work. Similarly we ca iterpret T () =T (/4) + 2 as i order to solve a problem of size we must solve 1 problems of size /4 ad do 2 uits of additioal work. We ca also iterpret the recurrece T () =3T ( 1) + as i order to solve a problem of size, we must solve 3 subproblems of size 1 ad do additioal uits of work. We will ow draw the begiig of the recursio tree for (4.16). For ow, assume is a power of 2. The tree has three parts. O the left, we keep track of the problem size, i the middle we draw the tree, ad o right we keep track of the work doe. So to begi the recursio tree for (4.16), we take a problem of size, split it ito 2 problems of size /2 ad ote o the right that we do uits of work. The fact that we break the problem ito 2 problems is deoted by drawig two childre of the root ode (level 0). This gives us the followig picture.

3 4.3. GROWTH RATES OF SOLUTIONS TO RECURRENCES 83 /2 We the cotiue to draw the tree i this maer. Addig a few more levels, we have: /2 /2 + /2 = /4 /4 + /4 + /4 + /4 = /8 8(/8) = We see that at each level, we halve the problem size ad double the umber of subproblems. We also see that at level 1, each of the two subproblems requires /2 uits of additioal work, ad so we get a total of work. Similarly the secod level has 4 subproblems of size /4 adso we get 4(/4) uits of additioal work. We ow have eough iformatio to be able to draw the tree i geeral. To do this, we eed to determie, for each level i, threethigs umber of subproblems, size of each subproblem, total work doe. We also eed to figure out how may levels there are i the recursio tree. We see that for this problem, at level i, wehave2 i subproblems of size /2 i. Further we have (2 i )[/(2 i )] = uits of work per level. To figure out how may levels there are i the tree, we

4 84 CHAPTER 4. INDUCTION, RECURSION AND RECURRENCES just otice that at each level the problem size is cut i half, ad the tree stops whe the problem size is 1. Therefore there are log levels of the tree. We ca thus draw the whole tree as follows: log levels /2 /4 /2 + /2 = /4 + /4 + /4 + /4 = /8 8(/8) = 1 (1) = The importat thig is that we ow kow exactly how may levels there are, ad how much work is doe at each level. Oce we kow this, we ca sum the total amout of work doe over all the levels, ad this is the solutio to our recurrece. I this case, there are log + 1 levels, ad at each level the amout of work we do is uits. Thus we coclude that the solutio to recurrece 4.16 is T () =(log +1). Let s look at oe more recurrece. T () = T (/2) + if >1 1 if =1 (4.17) Agai, assume is a power of two. We ca iterpret this as, to solve a problem of size, we must solve oe problem of size /2 ad do uits of additioal work. Let s draw the tree for this problem: log levels /2 /4 /2 /4 /8 /8 1 1

5 4.3. GROWTH RATES OF SOLUTIONS TO RECURRENCES 85 We see here that the problem sizes are the same as i the previous tree. The rest, however, is differet. The umber of subproblems does ot double, rather it remais at oe o each level. Cosequetly the amout of work halves at each level. Note that there are still log levels, as the umber of levels is determied by how the problem size is chagig, ot by how may subproblems there are. So o level i, we have 1 problem of size /2 i, for total work of /2 i uits. We ow wish to compute how much work there is. Note that it is differet o each level, so we have that the total amout of work is which is a geometric series. + /2+/ =( ( 1 2 ) log2 ), From our formuala for the sum of a fiite geometric series i Corollary we see that if we have b equal to some umber r less tha 1, the r k approaches 0 as k gets large, so that k r i r i 1 1 r. Note that we are oly upper boudig the sum; however, our upper boud oly differs from the exact solutio by roughly r k ad for most problems we are cocered with this quatity is egligible. Applyig this to our equatio above we get that, ( ) 1 i 2 so T () =O(). ( ) 1 i /2 =2, Note that what we have doe is fid that all solutios T() satisfy T () =O(). We have ot actually foud a solutio. However for the kids of recurreces we have bee examiig, oce we kow T(1) we ca compute T () for ay by repeatedly usig the recurrece, so there is o questio that solutios do exist. What is ofte importat to us i applicatios is ot the exact form of the solutio, but a big-oh upper boud o the solutio Solve the recurrece T () = 3T (/3) + if 3 1 if <3 usig a recursio tree. Assume that is a power of Solve the recurrece T () = 4T (/2) + if 2 1 if =1 usig a recursio tree. Assume that is a power of Ca you give a geeral Big Oh formula for solutios to recurreces of the form T () =at (/2) + whe is a power of 2? You may have differet aswers for differet values of a.

6 86 CHAPTER 4. INDUCTION, RECURSION AND RECURRENCES The recurrece i Exercise is similar to the mergesort recurrece. The differece is that at each step we divide ito 3 problems of size /3. Thus we get the followig picture: /3 /3 + /3 + /3 = log levels /9 9(/9) = 1 (1) = The oly differece is that the umber of levels, istead of beig log 2 +1isowlog 3 +1, so the total work is still O( log ) uits. Now let s look at the recursio tree for Exercise Now, we have 4 childre of size /2, adwegetthefollowig: /2 /2 + /2 + /2 + /2 = 2 log levels /4 16(/4) = 4 1 ^2(1) = ^2 Let s look carefully at this tree. Just as i the mergesort tree there are log levels. However, i this tree, each ode has 4 childre. Thus level 0 has 1 ode, level 1 has 4 odes, level 2 has 16 odes, ad i geeral level i has 4 i odes. O level i each ode correspods to a problem of size /2 i ad hece requires /2 i uits of additioal work. Thus the total work o level i is 4 i (/2 i )=2 i uits. Summig over the levels, we get log 2 log 2 2 i = 2 i.

7 4.3. GROWTH RATES OF SOLUTIONS TO RECURRENCES 87 There are may ways to evaluate that expressio, for example from our formula for the sum of a geometric series we get. log 2 2 i = 1 2(log 2 ) = =2 2. If we were iterested oly i a big-oh boud o solutios to our recurrece, we could write ( ) = ( + / ) = ((1 + 1/2+1/4+ +1/)) ((1 + 1/2+1/4+ )) ((2)) = 2 2. Thus ay solutio to our recurrece is O( 2 ). This gives a pretty good upper boud to the exact formula above. This kid of approximatio sometimes works i cases where we do t have a exact formula as we did above. Now let s compare the trees for the recurreces T () =2T (/2) +, T () =T (/2) + ad T () =4T (/2) +. Note that all three trees have depth log 2,asthisisdetermiedby the size of the subproblems relative to the paret problem, ad i each case, the sizes of the subproblems are 1/2 the size of of the paret problem. To see the differeces, i the first case, o every level, there is the same amout of work. I the secod case, the amout of work decreases, with the most work beig at level 0. I fact, it decreases geometrically, so the total work doe is a costat times the work doe at the root ode. I the third case, the umber of odes per level is growig at a faster rate tha the problem size is decreasig, ad the level with the largest work is the last oe. Agai we have a geometric series, ad the total work is at most a costat times the amout of work doe at the last level. If you uderstad these three cases ad the differeces betwee them, you ow uderstad the great majority of the recursio trees that arise i algorithms. So to aswer Exercise 4.3-3, a geeral Big Oh form for T () =at (/2) +, we ca coclude the followig: 1. if a<2thet () =O(). 2. if a =2theT () =O( log ) 3. if a>2thet () =O( log 2 a ) Cases 1 ad 2 follow immediately from our observatios above. We ca verify case 3 as follows. At each level i we have a i odes, each correspodig to a problem of size /2 i.thusat level i the total amout of work is a i (/2 i )=(a/2) i uits. Summig over the log 2 levels, we get (a/2) i. log 2 Sice a>2 we kow that we have a geometric series, so the sum will be at most a costat times the largest term. (See Lemma 4.2.1) The largest term i this case is clearly the last oe, amely (a/2) log 2, ad applyig rules of expoets ad logarithms, we get that the largest term is ( ) a log2 = alog log 2 = 2log2 a log2 2 log 2 = log2 a = log 2 a

8 88 CHAPTER 4. INDUCTION, RECURSION AND RECURRENCES Thus the whole sum is O( log 2 a ). Master Theorem We ca geeralize the previous result to hadle somewhat more geeral recurreces. There is o reaso that the amout of work doe by each subproblem eeds to be the size of the subproblem. I may applicatios it will be somethig else, ad we wish to geeralize our formula to hadle this. We will geeralize it to be c for some costat c. Similarly, the subproblems do t have to be 1/2 the paret problem. We the get the followig theorem. Theorem Let a be a iteger greater tha or equal to 1 ad b be a real umber greater tha or equal to 1. Letc be a positive real umber. Give a recurrece of the form the for a power of b, T () = at (/b)+ c if >1 O(1) if =1 1. if log b a<c, T () =O( c ) 2. if log b a = c, T () =O( c log ) 3. if log b a>c, T () =O( log b a ) Proof: Let s thik about the recursio tree for this recurrece. There will be log b levels. At each level, the umber of subproblems will multiply by a, ad so the umber of subproblems at level i will be a i. Each subproblem at level i correspods to a problem of size (/b i ). A subproblem of size /b i requires (/b i ) c additioal work ad sice there are a i of them, the total umber of uits of work o level i is ( ) a a i (/b i ) c = c i ( ) a i = c b c. b ci Recall from above that the differet cases (for c = 1) were whe the work per level was costat, decreasig or icreasig. Sice the work o level i is depedet o somethig to the ith power, this depeds o whether that somethig is 1, less tha 1 or greater tha 1. Thus our three cases deped o the value of ( a b c ) relative to 1. Now observe that ( a b )=1 c b c = a c log b =loga c =log b a Thus we see where our three cases come from. Now we proceed to show the boud i the differet cases.

9 4.3. GROWTH RATES OF SOLUTIONS TO RECURRENCES 89 I case 1, we have that log b c ( a log b b c )i = c ( a b c )i Sice this is a geometric series with a ratio of less tha 1 we ca upper boud this by c ( a b c )i = c 1 ( 1 a ). b c But a, b ad c are costats, so this is just O( c ) Prove Case 2 of the Master Theorem Prove Case 3 of the Master Theorem. I Case 2 we have that a b c =1adso log b c ( ) a i log b b c = c 1 i = c (1 + log b )=O( c log ) I Case 3, we have that a b c > 1. So i the series log b c ( a log b b c )i = c ( a b c )i, the largest term is the last oe, ad we ca, usig Lemma?? boud this sum by a costat times Thus the solutio is O( log b a ). c ( a b c )log b = c a log b (b c ) log b = c log b a log b b c = c log b a c = log b a We ote that the proof actually works for ay real umber a>1, but we do ot give the details here. Exercises E4.3-1 Draw recursio trees ad evaluate the followig recurreces (do ot use the master theorem). For all of these, assume that T (1) = 1 ad is a power of the appropriate iteger.

10 90 CHAPTER 4. INDUCTION, RECURSION AND RECURRENCES 1. T () =8T (/2) + 2. T () =8T (/2) T () =3T (/2) + 4. T () =T (/4) T () =3T (/3) + 2 E4.3-2 Recursio trees will still work, eve if the problems do ot break up geometrically, or eve if the work per level is ot c uits. Draw recursio trees ad evaluate the followig recurreces (do ot use the master theorem). For all of these, assume that T (1) = T () =T ( 1) + 2. T () =2T ( 1) + 3. T () =T ( )+1 4. T () =2T (/2) + log (Assume is a power of 2.) E4.3-3 Show that the sum of a geometric series k r k with r>1is(o(r k ), where the costat i the big-o depeds oly o k. Whatkid of big-oh statemet ca you make if r =1? Whatifr<1? E4.3-4 If S() =as( 1)+g() adg() <c with 0 c<a,howfastdoess() grow? E4.3-5 Use the master theorem to solve the recurreces i problem 1.