Hand Out: Analysis of Algorithms. September 8, Bud Mishra. In general, there can be several algorithms to solve a problem; and one is faced

Transcription

1 Had Out Aalysis of Algorithms September 8, 998 Bud Mishra c Mishra, February 9, 986 Itroductio I geeral, there ca be several algorithms to solve a problem; ad oe is faced with the problem of choosig a algorithm for use Oe way to dieretiate betwee two competig algorithms is to assig a complexity measure to the algorithms Havig doe this, oe may choose the algorithm with the lowest complexity There are two kids of complexity measures Static A complexity measure is static, if it does ot deped o the iput values A example of such a measure is the program legth This measure is relevat oly if the algorithm is to be used oce or oly a few times Dyamic A complexity measure is dyamic, if it varies with the iput values Some examples of such a measure are ruig time or storage space complexity of a algorithm as a fuctio of the iput values This measure is relevat, whe the algorithm is to be ru several times o `large' iput values I most practical applicatios, the dyamic complexity measure is sigificat, sice all productio softwares are assumed to ru a large umber of times, ad the cost of producig ad maitaiig the software (which deped o static complexities) are amortized rather fast We shall use ruig time as our complexity measure, sice almost all the algorithms we cosider have a space boud that is a liear fuctio of the iput size Furthermore, while aalyzig ruig times, we will igore costat factors, ad will cocetrate oly o the orders of growth There are two reasos for doig so This allows us to igore details of the machie model, (such as, the hardware of the machie, the istructio sets of the computer, the memory structure of the computer, the quality of code geerated by the compiler, etc) thus givig us a machie-idepedet complexity measure

2 For large eough problem sizes the relative eciecies of two algorithms deped o the ruig times as a asymptotic fuctio of iput size, idepedet of costat factors We shall geerally measure the ruig time, T (), as a fuctio of the worst-case iput data of size ; that is, T () is the maximum, over all iputs of size, of the ruig time o that iput The worst-case aalysis provides a performace guaratee, but may be overly pessimistic, if the worst-case iputs occur seldom A alterative is a average-case aalysis we measure the ruig time, T avg (), as the average, over all possible iputs of size, of the ruig time o that iput However, such a aalysis is frequetly mathematically itractable Furthermore, we must take care that our probability distributio is realistic which may be much harder to accomplish Faced with these problems, algorithm-desigers have suggested other averagecase complexity measures; but these measures have ot achieved wide-spread acceptace, ad requires mathematics, outside the scope of this class Hece, we shall restrict ourselves to worst-case complexity aalysis, ad occasioally, try to do the average-case aalysis Big Omicro, Big Omega, Big Theta We talk about the orders of growth of the fuctio T () usig the fuctios O() (kow as big-omicro or more popularly, big-oh), () (big-omega) ad () (big-theta) They are deed as follows O(f()), if there are two positive costats C ad 0 such that T () C f(); for all 0 (f()), if there are two positive costats C ad 0 that T () C f(); for all 0 such Some dee this dieretly (f(), if there is a positive costat C such that T () C f() iitely ofte (for iitely may values of ) This is a weaker deitio, ad seems more useful for lower-boud proofs but, for all practical purposes, the stroger deitio works pretty well

3 (f()), if there are three positive costats C, C 0 ad 0 such that C f() T () C 0 f(); for all 0 3 Algebra o O May of the rules of algebra work with O-otatios, but some do't Here, the mai problem is that, i the expressio O(f()), `=' is a oe-way equality (ie, it does ot have the symmetry property of the equality) For istace + = O( ) but ot O( ) = + (Why ot?) So how do you maipulate expressios with O's? Very carefully Here are some of the simple operatios you ca do with the O-otatio f() = O(f()); c O(f()) = O(f()); c is a costat; O(f()) + O(f()) = O(f()); O(O(f()) = O(f()); O(f(()) O(g()) = O(f() g()); O(f() g()) = f() O(g()) Usig these, we ca derive may other useful relatios Let T () = O(f()) be the ruig time of the program P, ad T () = O(g()), the ruig time of P The followig program P procedure P (N); begi P (N); P (N) ed has ruig time T ()+T () It is easy to show that f() = O(max(f(); g())) ad g() = O(max(f(); g())) Usig the above idetities(?), we see that T () + T () = O(f()) + O(g()) = O(O(max(f(); g()))) + O(O(max(f(); g()))) = O(max(f(); g())) + O(max(f(); g())) = O(max(f(); g())) 3

4 4 Aalysis of Algorithms ad Recurrece Relatios I geeral, we are iterested i aalyzig algorithms that is made of a set of a mutually recursive algorithms No-recursive algorithms are usually easy to aalyze, ad require very little mathematical sophisticatio I what follows, we will cocetrate o self-recursive algorithms, sice these illustrate all the mai ideas, ad geeralizatio to mutually recursive algorithms is straightforward Let us write self-recursive algorithm i the followig schematic form procedure P (N); begi Statemet ; P (N ); Statemet ; P (N ); Statemet 3; Statemet l; P (N l ); Statemet (l + ); ed A geeral program may be trasformed ito this form after resolvig ifthe-else's ad urollig while-loop's Hece, l itself ca be fuctio of Size(N) Such schemas are somewhat simplistic, but ot overly restrictive Let Size(N) =, Size(N ) =,, Size(N l ) = l Assume that the time spet by Statemet, Statemet,, Statemet (l+) is bouded from above by f() The we ca write the complexity of the algorithm as follows T () lx i= T ( i ) + f() Such a iequality is called a recurrece relatio What we preset here are, i fact, techiques to solve recurrece equatios 4

5 of the form lx i= T ( i ) + f() Uder a rather mild assumptio, you ca show that replacig the iequality by a equality does ot create ay problem (This will be a home work problem) 5 Solvig the Recurrece Equatios The subject of solvig recurrece equatios (also called dierece equatios) arise i may other areas too combiatorics, probability theory, discretetime cotrol theory, ecoomics to ame a few There are several powerful mathematical techiques available to solve these such as, summig factors, geeratig fuctios, z-trasformatios, operator methods, etc But most of these techiques are beyod our scope We develop oly a few techiques though simple, they are suciet for our purpose 5 Guess-Work The idea is to guess a solutio, ad the verify the guess by a mathematical iductio over the iteger This method requires you to kow the solutio i order to get the solutio, ad does ot work Use it oly whe you have o other way of solvig the equatio try several guesses; if you are lucky, oe of them will work Example 5 Cosider the followig recurrece equatio ( c ; if = ; T ( ) + c ; if > () Suppose we guess that c lg +c, the by mathematical iductio over all positive itegers, we ca show the validity of our guess Hece O( lg ) It is left as a exercise for you to do the steps of the iductive proof 5

6 5 Expasio Aother way to solve a recurrece relatio is to repeatedly expad the terms as may times as possible Whe this process stops, we will be left with a sum cosistig of fuctios of f() ad lower values of T () I may cases the summatio ca be evaluated to obtai a close form solutio This method is rather messy, ad proe to error avoid it, i all but a few simple cases Example 5 Cosider the followig recurrece equatio ( c ; if = ; T ( ) + c ; if > () First rearrage the recursive part as c + T Expadig the recursive deitio of T ( ) shows that c + Multiplyig ad cacelig yields c + T 4 c + c + 4 T 4 Applyig the recursive deitio of T ( ) to above shows that 4 c + c + c + 8 T 8 This process of expasio ad cacellatio ca be iterated lg times to yield X 4 lg i= 3 c 5 + T () = c lg + c 6

7 53 Recursio Tree Aother way to solve the recurrece equatio of the form T (0 ); if = 0 ; T ( ) + T ( ) + + T ( l ) + f(); if > 0, (3) is by expadig the terms of it i a tree like maer The tree correspodig to this expasio is called a recursio tree, ad is deed as follows The recursio tree for the equatio 3 at the value is a sigle ode with the value T ( 0 ), if = 0 ; otherwise, a ode with the value f() ad l sos such that the i th so is the root of a recursio tree of the equatio 3 at i It is easy to see that T () is just the sum of values at the odes Example 53 Cosider the followig recurrece equatio ( c ; if = ; T ( ) + c ; if > (4) The recursio tree for the equatio 4 is as show below C C C C C C depth = lg + Figure Recursio Tree You may prove the followig facts about the above recursio tree 7

8 The tree has a depth of lg + The sum of values at all the iteral odes at level i (0 i < lg ) is c Thus X j=iteral ode value(j) = c lg The tree has leaves The value of each leaf ode is c X j=leaf ode value(j) = c Hece X j=ode value(j) = c lg + c 54 Telescopig The idea of telescopig or summig factors is rather simple, ad is developed step-by-step i this ad the ext sectio We start out by showig how to solve recurrece equatios of very simple form usig this techique But this techique, whe combied with the trasformatio techiques of the ext sectio, is quite powerful ad hadles almost all equatios Please try to master this techique by tryig to solve several recurrece equatios this is the ocial techique for this course Cosider the recurrece equatio ; if = 0; T ( ) + ; if (5) We ca write dow the above equatio as follows T () T ( ) = T ( ) T ( ) = T () T (0) = T () T (0) = The simplicatio is obtaied by cacelig T (i) of the curret lie by the T (i) of the ext lie Thus whe we added up all the equatios most of the 8

9 terms caceled out We say the sum telescopes Note that the al aswer is + T (0) = + Fidig the solutio above was almost trivial Sometimes however we have to use a little trickery to make the sum of successive equatios telescope Suppose that our recurrece is a 0 T (0) = c 0 ; a b T ( ) + c ; if (6) where a, b ad c are give If we try to add to successive copies of this equatio, for example, a T () b T ( ) = c a T ( ) b T ( ) = c the the T ( ) terms do ot cacel sice b is ot equal to a Note however that we could multiply the equatios by some factor which would make the T ( ) terms cacel ad the sum will telescope Let us multiply the th equatio by some (so far uspecied) factor f to obtai f a T () f b T ( ) = f c f a T ( ) f b T ( ) = f c I order to guaratee that the T ( ) terms cacel we require that or f b = f a f = f a By urollig the above equatio we get f = f a Fixig f 0 =, we get b b! a a = f b b Q = = f a i=0 i 0 Q b i= i f 0 = ; Q i=0 f = a i Q ; if (7) i= b i 9

10 f is called the summig factor for the equatio 6 Let us write Sice this yields R() = f a T () f a f b T ( ) + f c = f a T ( ) + f c ; Now the telescopy will work Hece ad simplifyig, R(0) = c 0 ; R() = R( ) + c f ; if R() R( ) = c f R( ) R( ) = c f R() R(0) = c f R() R(0) = P j= c jf j R() = f a 4 c0 + 4 c0 + f a Q! b i= i Q 4 a c0 + i=0 i X j= X j= X j= c j f j 3 5 c j f j 3 5 ; Q j c j Q j i= b i i=0 a i Example 54 Cosider the followig recurrece equatio 3 5 (8) ; if = 0; T ( ) + ; if (9) This is same as T (0) = ; T () T ( ) = ; if 0

11 This is a example of 6 with By 7 we may choose a = ; b = f 0 = ; f = ; if Multiplyig by f ( ) gives the ew equatio Usig telescopy we get T (0) = ; T () T () = ; if Hece T () T ( ) = T ( ) T ( ) = T () = T () T (0) = ( + ) 55 Rage ad Domai Trasformatios Now, we examie how to trasform a sequece so that the trasformed sequece becomes more recogizable more familiar The fuctio T maps the iteger i its domai to a real T () i its rage We call a trasformatio o the values of the sequece T () a rage trasformatio ad a trasformatio o the idices a domai trasformatio We illustrate the ideas usig examples Domai Trasformatio Example 55 Cosider the followig recurrece equatio ( c ; if = ; T ( ) + c ; if > (0)

12 Let us use the followig domai trasformatio rst = k ; (k = lg ) hece S(k) = T ( k ) The above equatio becomes c ; if k = 0; S(k) = S(k ) + c k ; if k Now we ca use telescopig to get the followig solutio S(k) = T ( k ) = k (c k + c ) Back-substitutig for k ad lg for k, we coclude that Rage Trasformatio (c lg + c ) Example 56 Cosider the followig (pseudo-o-liear) recurrece equatio ; if = 0; () 3 (T ( )) ; if Let us use the followig rage trasformatio rst S() = log 3 T (); 3 S() = T () We may rewrite the recurrece as 0; if = 0; S() = S( ) + ; if Usig multiplier f =, we get Usig telescopy, we get Hece, Hece, S() S(0) 0 = S(0) = 0 0 S() S( ) = X = i i= (=) (=)+ (=) S() = 3 S() = 3 =

13 6 Examples 6 Example Divide-ad-Coquer A importat class of recurrece equatios results from algorithms based o the `divide-ad-coquer' paradigm; such algorithms cosist of three steps divide Break the problem of size ito a smaller subproblems each of size, usig o more tha g() time b recur Recursively solve each of the a subproblems usig a T ( b ) time 3 marry Combie the solutios to the subproblems to get a solutio to the origial problem, usig o more tha h() time The recurrece equatio is of the form a + f(); where f() = g() + h() () b Use the followig domai-trasformatio rst = b m (m = log b ) The above equatio becomes T (b m ) = a T (b m ) + f(b m ) Usig the multiplyig factor f m = ( a m ) we get Usig telescopy T (b m ) a m = T (bm ) a m + f(bm ) a m T (b m ) a m T (bm ) = f (bm ) a m a m T (b m ) T (bm ) = f (bm ) a m a m a m T (b) T () = f (b) a a T (b m ) a m T () = P m i= 3 f (b i ) a i

14 Hece T (b m ) = a m "T () + mx i= # f(b i ) a i assumptio f is a multiplicative fuctio; that is, f has the followig property f(m ) = f(m) f() Example of a multiplicative fuctio is f() = c Hece f(b i ) = (f(b)) Rewrite the previous fuctio as T (b m ) = a m 4 T () + m X i= f(b) a! i 3 5 Let The a m "T () + = mx i= There are three cases to cosider f(b) a! # i = a m T () + m case a > f(b); that is, < The a m O() = O (a m ) = O a log b = O a log b case a = f(b); that is, = The a m O(m) = O (a m m) = O a log b log b case3 a < f(b); that is, > The a m O ( m ) = O (f(b) m ) = O = O log b a log b f(b) log b = O log b f (b) 4

15 6 Example Radomized Divide-ad-Coquer Let us cosider a radomized algorithm (ie algorithm ivolvig coitosses), also based o the `divide-ad-coquer' paradigm; such a algorithm cosists of three steps radom-divide Break the problem of size ito smaller subproblems as follows rst choose a umber i ( i ) with probability ; let the rst subproblem be the oe cosistig of the rst i elemets, ad the secod, cosistig of the remaiig i elemets The divisio step takes o more tha g() amout of time recur Recursively solve each of the subproblems usig T avg (i) + T avg ( i) time 3 marry Combie the solutios to the subproblems to get a solutio to the origial problem, usig o more tha h() time Let us assume that g() + h() = c Hece the recurrece equatio will be T avg () = X i= This ca be rewritte as T avg () = = = = [T avg(i) + T avg ( i)] + c P P P i= (T avg (i) + T avg ( i)) + c T i= avg(i) + P T i= avg( i) + c P i= T avg(i) + c T i= avg(i) + P T j= avg(j) + c Or ( ) T avg () = Substitutig + for, we get T avg ( + ) = X i= X i=! T avg (i) + c ( ) (3)! T avg (i) + c ( + ) (4) 5

16 Subtractig equatio 3 from equatio 4, T avg ( + ) ( ) T avg () = T avg () + c Simplifyig the above equatio T avg ( + ) = ( + ) T avg () + c Or T avg ( + ) + Usig telescopy we get T avg() = c + T avg(+) Tavg() + T avg() Tavg( ) = c + = c T avg() Tavg() = c T avg(+) + T () = c Hece Or, T avg ( + ) = ( + ) [T avg () + c (H + )] T avg () = [T avg () + c (H )] o The approximate size of the th harmoic umber H is a well-kow quatity H = l + + O Here = is Euler's costat Hece T avg () = h c l + + O ( ) + T avg () c = ( l )c lg + (c( ) + T avg ()) + c O( ) = 39c lg + [T avg () 085c] + c O( ) = O( lg ) i 6

17 7 Ed Notes These otes are about years old I origially wrote them for the master's level graduate course i algorithms ad had derived these otes from my earlier otes from graduate school, a survey paper by Leuker ad ideas due to Moier, Betley ad Saxe Clearly, there are some ew iterestig ideas that have evolved i the iterveig years The readers are urged to cosult these more up-to-date results 7