Itroductio The focus of this module is mathematical aspects of algorithms. Our mai focus is aalysis of algorithms, which meas evaluatig efficiecy of algorithms by aalytical ad mathematical methods. We start by some simple examples of worst-case ad average-case aalysis. We the discuss the formal defiitios for asymptotic complexity of algorithms, used to characterize algorithms ito differet classes. Ad we preset examples of asymptotic aalysis of algorithms. The ext module deals with recursive algorithms, their correctess proofs, aalysis of algorithms by recurrece equatios, ad algorithmic divide-ad-coquer techique. Cotets Worst-case ad Average-Case Aalysis: Itroductory Examples Sequetial Search Fidig Max ad Mi Defiitios of Asymptotic Complexities O( ), Ω ( ), ad Θ( ) Sum Rule ad Product Rule Example of Aalysis: A Nested Loop Isertio Sort Algorithm Worst-Case Aalysis of Isertio Sort Average-Case Aalysis of Isertio Sort 1
Worst-case ad Average-Case Aalysis: Itroductory Examples Sequetial Search Algorithm Suppose we have a array of elemets A[1: ], ad a key elemet, KEY. The followig program is a simple loop which goes through the array sequetially util it either fids the key, or determies that it is ot foud. (dtype is the type declaratio for the array.) it SequetialSearch (dtype A[ ], it, dtype KEY) { for i = 1 to { if (KEY == A[i]) retur (i); }; retur ( 1); //The for-loop eded ad the key was ot foud. } Let us aalyze the umber of key-comparisos (highlighted) i this algorithm. By coutig this domiat operatio, basically we are coutig the umber of times the loop is executed. There are two commo ways of doig the aalysis: Worst-case aalysis: This determies the maximum amout of time the algorithm would ever take. This aalysis is usually easier tha the average-case aalysis. At the same time, it is hoped that it is a good reflectio of overall performace. Average-case aalysis: This method determies the overall average performace. It cosiders all possible cases, assigs a probability to each case, ad computes the weighted average (called expected value) for the radom variable (i this case, the umber of key-comparisos). Worst-Case Aalysis of Sequetial Search The worst-case umber of key-compariso i this algorithm is obviously. This happes if the key is foud i the last positio of the array, or if it is ot foud aywhere. Sice each iteratio of the for-loop takes at most some costat amout of time, C, the the total worst-case time of the algorithm is T() C + D. (The costat D represets the maximum amout of time for all statemets that are executed oly oce, idepedet of the variable. ) This total time is characterized as order of, deoted as O(). (We will shortly see the formal defiitio for the order.)
Average-Case Aalysis of Sequetial Search Now let us compute the average umber of key-comparisos. As a first estimate, oe may thik that sice the worst-case umber is, ad the best-case is 1 (foud right away), the the average must be about /. Let us do a careful aalysis ad see how good this estimate is. First, as a quick review the expected value, suppose a radom variable has the possible values {1,,3} with the followig probabilities. Value of the radom variable r Probability P r 1 0.1 0.1 3 0.8 The the expected value of r is 1 0.1 + 0.1 + 3 0.8 =.7 We may also refer to this as the weighted average. Note that a straight average (whe there is o probability ivolved) would be simply (1 + + 3)/3 =. Now, to compute the expected value of the umber of key-comparisos for the algorithm, let ad let P = Probability that the key is foud somewhere i the array P i = Probability that the key is foud i positio i of the array, 1 i. Assumig that the array is radom, a commo assumptio is that whe the key is foud, the it is equally likely that it is foud i ay of the positios. So, P i = P, i. Fially, the probability that the key is ot foud is Q = 1 P 3
So, the expected umber of key-comparisos i this algorithm is: f() = P i i + Q = P i + (1 P) = P i + (1 P) (Use arithmetic sum formula) = P ( + 1) + (1 P) = P + 1 + (1 P) I the special case whe P = 1, the expected umber of comparisos is +1, which agrees with our iitial estimate. Otherwise, there is a additioal term that takes ito accout the additioal evet whe the key is ot foud aywhere i the array. Fidig Max ad Mi of a Array The followig pseudocode is a simple program loop that fids the maximum ad miimum elemets i a array of elemets, A[1: ]. (Max ad Mi are the retured parameters.) FidMaxMi (dtype A[ ], it, dtype Max, dtype Mi) { Max = A[1]; Mi = A[1]; for i = to { if (A[i] > Max) Max = A[i]; else if (A[i] < Mi) Mi = A[i]; } I iteratio i of the for-loop, A[i] is first compared agaist Max. If A[i] is greater, the Max is updated. Otherwise, a secod compariso is made agaist Mi, ad if A[i] is smaller, the Mi is updated. Let us aalyze the worst-case, best-case, ad average-case umber of keycomparisos. (The key comparisos are highlighted.) 4
Worst-Case ad Best-Case Aalysis I the worst-case, every iteratio of the loop makes two comparisos. (This happes if the first elemet of the array has the largest value.) So the worst-case umber of comparisos is ( 1). I the best-case, every iteratio makes oly oe compariso, so the best-case umber of comparisos is ( 1). This happes if the iput is i sorted order. Average-Case Aalysis The umber of comparisos i each iteratio of the loop is i the worst-case, ad 1 i the best-case. So is it a good estimate to figure that o the average, the umber is 1.5 per iteratio?! No, the average is ot always half-way betwee the worst ad the best. (If I buy a lottery ticket, do I have a 50-50 chace of wiig the jackpot?!) We will prove that it is much more likely to make two comparisos per iteratio. As a result, the expected umber of comparisos is very close to the worst-case umber. Let us assume the array is radom ad the elemets are all distict. Iteratio i of the loop compares elemet A[i] agaist the curret Max ad possibly Mi. Let us defie the prefix sequece of A[i] as the sequece of elemets i the array startig with A[1] ad edig with A[i] itself. A[1], A[],, A[i] Sice the array is radom, elemet A[i] is equally likely to be the smallest i its prefix, or the secod smallest, or third smallest,, or the largest. So, the probability that A[i] is the largest i its prefix sequece is ad the probability that A[i] is ot the largest i its prefix sequece is 1 i i 1 i If A[i] is the largest i its prefix, iteratio i makes oly oe compariso. Otherwise, iteratio i makes two comparisos. Therefore, the expected umber of comparisos is f() = ( 1 i 1 + i 1 ) = ( 1 i i ) = ( 1) 1 i i= i= i= = 1 1 i The latter summatio is kow as the Harmoic series H, ad the value of the sum is approximately l. (l is the atural log.) 5
1 i H = l Therefore, f() 1 l. Sice the value of the log is egligible compared to, the expected umber of comparisos is ideed very close to the worst-case value, as stated earlier. For example, if = 1000, the expected umber of key-compariso is about 199, ad the worst-case umber ( ) is 1998. (Note: Homework problems explore a techique for fidig approximate value of a summatio by covertig the summatio ito itegral. This techique may be used to fid the approximate value for the Harmoic sum.) Defiitios of Asymptotic Complexity Whe we study the ruig time of a algorithm, our focus is o the performace whe the problem size gets large. This is because the performace for small problem sizes hardly matters sice the ruig time will be very small ayway. Suppose there are two algorithms for a problem of size with the ruig times, respectively T 1 () = 10, T () = Which oe of the two is faster (smaller) ruig time? Let s tabulate these two fuctios for some values of. 10 1 10 0 8 5 50 50 10 100 00 100 1,000 0,000 1,000 10,000,000,000 10,000 100,000 00,000,000 100,000 1 10 6 10 10 6
We observe that iitially, for < 5, T 1 is larger tha T. The two equal at = 5. Ad after that, as gets larger, T gets much larger tha T 1. This may also be observed pictorially, by lookig at the graphs of these fuctios (time-versus-). Time T (quadratic) T 1 (Liear, with slope of 10) The quadratic fuctio T starts smaller tha the liear oe T 1. The two cross at = 5. After that, T starts growig much faster tha T 1. As gets larger ad larger, T gets much larger tha T 1. We say that T has a faster growth rate tha T 1, or that T is asymptotically larger tha T 1. The fact that T 1 has a slower growth rate tha T is due to the fact that T 1 is a liear fuctio ad T is a quadratic fuctio. The coefficiets (also called costat factors) are ot as critical i this compariso. For example, suppose we have a differet pair of coefficiets: T 1 () = 50, T () = The cross poit betwee the two fuctios ow is = 50, ad after that T starts growig much faster tha T 1 agai. So, the asymptotic complexity defiitios icorporate two issues: 1. Focus o large problem size (large ), ad igore small values of.. Igore the costat coefficiets (costat factors). We are ow ready to make the formal defiitio. Defiitio 1: O( ) Upper Boud Suppose there are positive costats C ad 0 such that T() C f(), 0 The we say T() is O(f()). The O( ) is read as order of, or big oh of. 7
Example: Prove the followig fuctio is O( ). T() = 5 + 10 + 100 Proof: Ituitively, whe gets large, the total value of this polyomial is close to 5 because the remaiig terms become egligible i compariso. Now, let us formally prove that T() is O( ) by fidig costats C ad 0 such that T() C, 0. T() = 5 + 10 + 100 5 + 10 () + 100 ( ), 1 115, 1 This satisfies the defiitio ad proves T() is O( ). (Here C = 115 ad 0 = 1. ) But, to satisfy our ituitio, how do we fid the costat C closer to 5? By pickig a larger 0. Let s arbitrarily pick 100, so ( 100 ) 1. T() = 5 + 10 + 100 5 + 10 ( ) + 100 ( 100 100 ), 100 5.11, 100. Example: Prove the followig polyomial is O( 3 ). T() = 5 3 10 + 0 100 Proof: Let s arbitrarily pick 10, so that ( 10 ) 1. T() = 5 3 10 + 0 100 (discard egative terms) 5 3 + 0, 5 3 + 0 ( 10 ), 10 5. 3, 10. Next, we make the followig defiitio for the lower boud, which is symmetrical to O( ). Defiitio : Ω( ) Lower Boud Suppose there are positive costats C ad 0 such that T() C f(), 0 The we say T() is Ω(f()). 8
Example: Prove the followig polyomial is Ω( 3 ). T() = 5 3 10 + 0 100 Proof: We must show that T() C 3, 0 for some positive costats C, 0. Here, we eed to pick 0 carefully so that C becomes positive. T() = 5 3 10 + 0 100 (discard smaller positive terms) 5 3 10 100 5 3 10 ( 3 ) 100 ( 100 100 ), 100 4.8999 3, 100. Defiitio 3: Θ( ) Tight Boud Suppose there are positive costats C 1, C, 0 such that C 1 f() T() C f(), 0 That is, T() is both O(f()) ad Ω(f()). The, we say that T() is Θ(f()). Example: We proved the followig polyomial is both O( 3 )ad Ω( 3 ). Therefore, T() is Θ( 3 ). Example: Suppose T() = 5 3 10 + 0 100 T() 5 I this case, T() is Ω() ad O( ). But this fuctio does ot have a tight boud. Example: Prove the followig summatio is Θ( ), without the use of arithmetic sum formula, but rather by maipulatig the terms to fid the eeded upper boud ad lower boud. S() = 1 + + 3 + + 1. Prove O( ): S() = 1 + + + + + +. 9
. Prove Ω( ): I the above proof for the upper boud, we raised all terms to the largest term. If we try to mimic that approach ad lower all terms to the smallest term, we get S() 1 + 1 + + 1 =, which will ot give the desired lower boud. Istead, we will first discard the first half of the terms, ad the lower all terms to the smallest. S() = 1 + + + = i i i= i= 4 We proved 4 S(). Therefore, S() is Θ( ). Sum-Rule ad Product-Rule Sum Rule for O( ): Suppose T 1 () is O(f()) ad T () is O(g()). The, T 1 () + T () is O(f() + g()). Proof: we first express T 1 () ad T () i terms of the defiitios of O(): T 1 () C 1 f(), 1 T () C g(), For max { 1, }, the two iequalities will both hold. So, T 1 () + T () C 1 f() + C g(), max{ 1, } Cf() + Cg(), C = max{c 1, C ) C(f() + g()). (Note: The sum rule similarly applies to Ω ad to Θ. ) 10
Applicatio of sum rule: Suppose a program has two parts: Part 1, which is executed first, ad the Part. Part 1 : Time T 1 () suppose O() Part : Time T () suppose O( ) The, the total time is O( + ), which is O( ). Product Rule for O( ) If T 1 () is O(f()) ad T () is O(g()), the, T 1 () T () is O(f() g()). Proof: Similar to the above proof; left to the studet. (Note: The product rule similarly applies to Ω ad to Θ. ) A applicatio of the product rule is i ested loops, as i the followig example. Example of Aalysis: A Nested Loop Let s aalyze the ruig time of the followig program (ested loops). C = 0 for i = 1 to + 1 for j = i to 3 1 C = C + 1 Ier loop umber of times = (3 1) (i) + 1 = 3 i Method 1 (Detailed Aalysis): Let us fid the total umber of times the iermost statemet (C = C + 1) is executed. That is, fid the fial value for C. Let F() deote this fial cout. We fid this cout by the followig double summatio. +1 3 1 F() = 1 j=i 11
The ier summatio for j is 1 + 1 + + 1, so we fid the cout by: Upper limit of summatio lower limit of summatio + 1 = (3 1) (i) + 1 = 3 i. Ad the outer summatio is arithmetic sum, so we apply the formula for it. +1 3 1 F() = 1 j=i +1 = (3 i) (This is arithmetic sum) = ( + 1) (5 ) = ( + 1) = 5 + 3 (3 1) + (3 1) The total umber of times the iermost statemet is executed is O( ), which meas the total ruig time of the program is also O( ). Method (Loop Aalysis): I this method, we do t bother to fid the exact total umber of times the iermost statemet is executed. Rather, we aalyze the loops by usig the sum-rule ad product-rule. The ier loop is executed 3 i times. Sice the rage of i values is 1 to + 1, the we kow 3 i is O(). The outer loop is executed + 1 times, which is O(). Therefore, by the product rule, the total ruig time is O( ). 1
Isertio Sort Algorithm Suppose we have a array of elemets, A[0: 1]. Isertio sort starts by sortig the first two elemets. The the third elemet is iserted ito the sorted part, so that the first 3 elemets are sorted. The, the ext elemet is iserted ito the sorted portio, so that the first 4 elemets become sorted, ad so o. Let us first illustrate by a umerical example. [7] 5 6 4 3 Sort the first two [5 7] 6 4 3 Isert 6 ito [5 7] [5 6 7] 4 3 Isert ito [5 6 7] [ 5 6 7] 4 3 Isert 4 ito [ 5 6 7] [ 4 5 6 7] 3 Isert 3 ito [ 4 5 6 7] [ 3 4 5 6 7] Let s look at the details of each isertio phase. For example, let us see how the last isertio is carried out. At the start of this phase, the sorted part is [ 4 5 6 7] ad elemet 3 eeds to be iserted ito the sorted part. This is doe by a sequece of compare/swap operatios betwee pairs, startig at the ed with the pair [7 3]. 4 5 6 [7 3] Cm/Swp 4 5 [6 3] 7 Cm/Swp 4 [5 3] 6 7 Cm/Swp [4 3] 5 6 7 Cm/Swp [ 3] 4 5 6 7 Compare 3 4 5 6 7 We preset the pseudocode ext, ad the aalyze both the worst-case ad averagecase time complexity. Isertio Sort (datatype A[ ], it ) { //Iput array is A[0: 1] for i = 1 to 1 {// Isert A[i]. Everythig to the left of A[i] is already sorted. j = i; while (j > 0 ad A[j] < A[j 1]) {swap(a[j], A[j 1]); j = j 1}; }; } 13
Worst-Case Aalysis of Isertio Sort Method 1 (Fid the worst-case Exact Number of Operatios): Oe way to aalyze a algorithm is i terms of some domiat operatio, i this case a key compariso. This is a compariso betwee a pair of elemets i the array, which is highlighted i the above algorithm. By coutig this operatio, we are basically coutig the umber of times the ier loop (while loop) is executed. So, let f() be the worst-case umber of key comparisos i this algorithm The worst-case umber of key comparisos i the while loop is exactly i, because the loop starts with j = i ad i the worst case goes dow to j = 1. (Whe j = 0, the compariso is skipped.) Therefore, the worst-case umber of key-comparisos is: 1 f() = i = ( 1) = (Arithmetic sum formula was used to fid the sum.) So, we coclude that the worstcase total time of the algorithm is O( ). Method (Aalyze the Loops): The worst-case time of the ier while-loop is O(i), thus O(). This is because the rage of i values is 1 to 1. The outer loop (for loop) is executed O() times. By the product rule, the total worst-case time of the algorithm becomes O( ). Average-Case Aalysis of Isertio Sort We carry out the average case aalysis i two ways, oe i terms of the umber of key comparisos, ad the other i terms of the umber of swaps. Expected Number of Key-Comparisos: Cosider the while loop, where elemet A[i] is iserted ito the sorted part. If we assume the array is radom, the A[i] has equal probability of beig the largest, secod largest, third largest, ad so o. There are i + 1 cases, as listed i the table below. The table also shows the umber of keycomparisos made by the while loop for each case. Note that the last two cases both make the worst-case umber of key comparisos, which is i. 14
Evet If elemet A[i] is: 1 Largest Secod largest 3 Third Largest i 1 Third smallest i Secod smallest i + 1 Smallest Number of key comparisos made by the while-loop 1 3 i 1 i i Sice these (i + 1) cases all have equal probabilities, the expected umber of key comparisos made by the while loop is: (Sice the value of last term is small, i(i + 1) (1 + + 3 + + i) + i = + i i + 1 i + 1 i i+1 = i + i i + 1 < 1, the expected umber of key comparisos made by the while loop is about i/, which is half of the worst-case.) Therefore, the expected umber of key-comparisos for the etire algorithm is 1 ( i + i i + 1 ) = i + i = i + 1 < ( 1) 4 1 + ( 1) 1 ( 1) 4 1 + i i + 1 (The upper boud for the summatio is a fairly good approximatio. For example, for = 100, the value of the summatio becomes 95.) Expected Number of Swaps I the above table, we saw the last two cases both have i comparisos. This ouiformity resulted i a slight complicatio i the aalysis. This complicatio is avoided by aalyzig the expected umber of swaps. Let us agai look at the table of possible evets. 99 i i+1 Evet If elemet A[i] is: 1 Largest Secod largest 3 Third Largest i 1 Third smallest i Secod smallest i + 1 Smallest Number of SWAPS made by the while-loop 0 1 3 i 1 i 15
Sice all i + 1 evets have equal probability, the expected umber of swaps for the while loop is 0 + 1 + + + i i + 1 = i So the expected umber of swaps for the etire algorithm is 1 i = ( 1) Alterative Aalysis for the Expected Number of Swaps: Suppose the iput sequece is a radom orderig of itegers 1 to. This aalysis cosiders all! possible iput orderigs, cout the umber of swaps for each case, ad average it over all possible cases. (We hope this aalysis provides additioal isight o how the expected performace may be computed.) Defie a pair of elemets (A[i], A[j]) i the array to be iverted (or out-of-order) if (i < j) ad A[i] > A[j]. Ad defie the total umber of iversios i a array as the umber of iverted pairs. Oe way to cout the umber of iversios is: 1 Number of elemets to the left of A[i] which are greater tha A[i]. From this formulatio, it should be obvious that the umber of iversios i a sequece is exactly equal to the umber of swap operatios made by the isertio-sort algorithm for that iput sequece. Suppose there are k elemets i the origial iput sequece to the left of A[i] with values greater tha A[i]. The, at the start of the while loop for isertig A[i], these k elemets will all be o the rightmost part of the sorted part, immediately to the left of A[i], as depicted below. (Elemets < A[i]) followed by (k elemets > A[i]) Sorted Part A[i] Elemet to be iserted The A[i] has to hop over the k elemets i order to get to where it eeds to be i the sorted part, which meas exactly k swaps. 16
The followig table shows a example for = 3. There are! = 6 possible iput sequeces. The umber of iversios for each sequece is show, as well as the overall average umber of iversios. Iput Sequece Number of Iversios 1 1 3 0 1 3 1 3 1 3 1 4 3 1 5 3 1 6 3 1 3 Overall Average = 9/6 = 1.5 The iput sequeces may be partitioed ito pairs, such that each pair of sequeces are reverse of each other. For example, the reverse of [1 3 ] is the sequece [ 3 1]. The followig table shows the pairig of the 6 iput sequeces. Pairs of iput sequeces which are reverse of each other 1 3 3 1 1 3 3 1 1 3 3 1 Number of iversios 0 3 1 1 Average umber of iversios for each pair 1.5 1.5 1.5 Overall Average: 1.5 I geeral, if a pair of elemets is iverted i oe sequece, the the pair is ot iverted i the reverse sequece, ad vice versa. For example: I the sequece [1 3 ], the pair (3, ) is iverted I the reverse sequece, [ 3 1], the pair (, 3) is ot iverted. This meas that each pair of values is iverted i oly oe of the two sequeces. So each pair cotributes 1 to the sum of iversios i the two sequeces. Therefore, the sum of iversios i each pair of reverse sequeces is the umber of pairs i a sequece of elemets, which is ( 1). So, the average umber of iversios for each pair of sequeces is W() = ( 1) 4 17
The overall average umber of iversios is also W(). Therefore, the expected umber of swaps i the algorithm is this exact umber. Deducig the Expected Number of Comparisos It is iterestig to ote that from the expected umber of swaps, we may arrive at the expected umber of comparisos rather quickly. Every key-compariso made by the while loop is followed by a swap, except possibly the last compariso before the termiatio of the while loop. This happes if (j > 0 ad A[j] A[j 1]), which meas that the elemet beig iserted is ot the smallest i its prefix sequece. This has the probability i (i + 1), as idicted earlier. So, with this probability, there is a last compariso i the while loop which is ot followed by a swap. Therefore, the expected umber of comparisos is 1 W() + which is the same as earlier. i ( 1) = i + 1 4 1 + i < i + 1 ( 1) 4 + 1 18