COMPSCI 330: Desig ad Aalysis of Algorithms 1/19/2016 ad 1/21/2016 Divide ad Coquer Lecturer: Debmalya Paigrahi Scribe: Tiaqi Sog, Tiayu Wag 1 Overview This set of otes is orgaized as follows. We begi by itroducig a O( 2 ) sortig algorithm, which motivates the cocept of divide ad coquer by itroducig QuickSort. We the proceed by aalyzig three more examples: merge sort, bit-strig multiplicatio, polyomial multiplicatio ad a O()-time algorithm for the problem of selectio. We coclude by itroducig the master theorem method for solvig recurrece relatios. 1 2 Isertio Sort We ow ivestigate a simple sortig algorithm. Algorithm 1 Isertio sort 1: fuctio INS-SORT(A[1...],k) 2: for i 1 to do 3: INSERT(A[1,...,i 1],i) assume INSERT iserts A[i] ito sorted positio i A[1,...,i 1] ad takes O(i) time. 4: retur A[1,...,] A recurrece relatio for Algorithm 1 is We aalyze the ruig time by computatio tree: T () = T ( 1) + O(). O() O(-1) -1 O(-2) -2 We have 1 Some of the material i this ote is from previous otes by Samuel Haey, Yilu Zhou ad Nat Kell for this class i Fall 2014. 2-1
T () = O() + O( 1) + O( 2) +... + O(1) = O( 2 ) Next, we show that we ca desig a faster sortig algorithm by divide-ad-coquer techique. 3 QuickSort I the followig algorithm, A is the uderlyig array/list; r,l are rightmost (largest), ad leftmost (smallest) idices respectively. 3.1 The algorithm Algorithm 2 QuickSort(A, l,r) 1: m Pivot(A,l,r) 2: Swap(A, l, m) 3: QuickSort(A,l,m 1) 4: QuickSort(A,m + 1,r) The Swap(A,i, j) fuctio just swap the values A[i] ad A[ j]. The Pivot fuctio is described i the followig sectio. 3.2 Pickig a pivot poit A pivot poit is the poit from where we divide the QuickSort ito two subproblems. Below is a algorithm for pickig a pivot poit. Algorithm 3 Pivot(A, l,r) 1: i l, j r + 1, x A[i], 2: while i < j do 3: Repeat i + + util x A[i] 4: Repeat j util x A[ j] 5: Swap(A,i, j) 6: Retur j Therefore, the recurrece relatio for QuickSort is T () = O() + T (i) + T ( i) where i is the chose pivot poit. I the worst case, where i = 1 or i = 1, T () = T ( 1) + O() = O( 2 ). I the best case, where i = 2, T () = 2T ( 2 ) + O() = O(log). 3.3 Extesio Radomly pickig a pivot poit I practice, we ofte radomly pickig a pivot poit with the followig procedure iserted before pickig a pivot. The modified versio is ofte referred to as Radomized QuickSort. 2-2
Algorithm 4 R-Select(A, l,r) 1: i Radom(l,r) 2: Swap(A, i, l) Here the Radom(l, r) fuctio uiformly radomly geerates a iteger that is betwee l ad r. The aalysis of the radomized QuickSort is beyod the scope of this course. With this trick, the expected ruig time is O(log). 4 Mergesort 4.1 Itroductio Mergesort is a sortig algorithm with both worst-case ad average-case performace of O( log ). The algorithm is also a recursive divide-ad-coquer alillustrategorithm, but istead of usig a pivot to decide where to partitio our subproblems, mergesort always divides the array equally. Mergesort the recursively sorts these two subarrays ad the combies (or merges) them ito oe master sorted array. 4.2 Pseudocode 1 // Msort(A,i,j) will sort elemet A[i] through A[j] i A 2 Msort(A,i,j) 3 B=Msort(A,i,(i+j)/2) 4 C=Msort(A,(i+j)/2+1,j) 5 A=merge(B,C) 6 7 // Merge(B,C) assumes sorted arrays B ad C 8 // ad merges them ito oe big sorted array 9 Merge(B,C) 10 D=empty array 11 i=j=k=1 12 while(i< B ad j< C ) 13 if(b[i]<c[j]) the D[k]=B[i], i++ 14 else the D[k]=C[j], j++ 15 k++ 16 // At this poit all elemets of oe array should be copied to D 17 Copy all elemets of the other array ito D 18 retur D 4.3 Example As a example, cosider the followig array: 7 1 5 2 0-2. Mergesort will divide the array ito two subarrays 7 1 5 ad 2 0-2. The it will recursively sort these two arrays ad give back 1 5 7 ad -2 0 2. To merge these two arrays, we first compare 1 with -2. Because -2 is smaller, it is copied to the fial sorted array. The 1 is compared with 0 ad 0 is copied to the fial array. The 1 ad 2. This time 1 is smaller so 1 is copied to the fial array. Next 5 is compared with 2 ad 2 is copied to fial array. Now the secod array has bee used up so 5 ad 7 of the first array are copied to the fial array. The fial array is -2 0 1 2 5 7. 2-3
4.4 Performace For mergesort, the worst-case ad average-case performaces are the same as the algorithm does ot ivolve ay radom selectio or subarrays of variable legth. Lie 3 ad 4 of the pseudocode each takes T (/2) time ad the Mergeprocedure takes O( B + C ) time, where B ad C are the sizes of the array B ad C. The recurrece relatio is thus T () = 2T (/2) + O(). The computatio tree is like: O() /2 /2 O(2(/2)) O(2 k (/2 k )) log 2 levels We have T () = O(log 2 ). 4.5 Compariso to QuickSort QuickSort is i-place, thus cosumig less space. O the other had, QuickSort is less parallelizable, ad ca potetially suffer from greater time complexity. Both algorithms are widely used, ad i fact, they are both expected to achieve the best ruig time that a compariso-based sortig algorithm ca hope for, which is O(log). 5 Multiplyig Bit Strigs The problem is to multiply two -bit umbers x ad y. The straightforward way is to divide each umber ito two parts ad each part has 2 bit. x y x L y L x R y R /2 /2 We have x = x L 2 2 + xr y = y L 2 2 + yr xy = x L y L 2 + (x L y R + y L x R )2 2 + xr y R The ruig time T () = 4T ( 2 ) + O(). This recurrece relatio ca be solved by computatio tree. 2-4
O() /2 /2 /2 /2 O(2 1 ) O(2 k ) log 2 levels We have T () = (1 + 2 + 2 2 +...2 k +... + 2 log 2 ) = (2 log 2 +1 1) = O( 2 ) There is a smarter way to do the multiplicatio based o the fact that xy = x L y L 2 + (x L y R + y L x R )2 2 + xr y R = x L y L 2 + ((x L + x R )(y L + y R ) x R y R x L y L ) + x R y R Oly three multiplicatios are eeded: x L y L, x R y R ad (x L + x R )(y L + y R ). We have T () = 3T ( 2 ) + O() The computatio tree is like O() /2 /2 /2 O((3/2)) O(3/2) k ) log 2 levels We have T () = (1 + 3 2 + (3 2 )2 +...( 3 2 )k +... + ( 3 2 )log 2 ) = O(( 3 2 )log 2 ) = O(3 log 2 ) = O( log 2 3 ) O( 1.585 ) 6 Polyomial Multiplicatio Polyomial Multiplicatio is aother example where Divide-ad-Coquer techique ca be applied. Assume ad P(x) = Q(x) = k=0 k=0 a k x k b k x k. 2-5
If we aively multiply them term by term, it would be of complexity O( 2 ). However, we ca apply the followig trick. Write ad The P(x) = Q(x) = 2 i=0 2 i=0 a i x i + x 2 b i x i + x 2 2 i=0 a i+/2 x i = P 1 (x) + x 2 P2 (x) 2 b i+/2 x i = Q 1 (x) + x 2 Q2 (x). i=0 PQ = P 1 Q 1 + x 2 (P1 Q 2 + P 2 Q 1 ) + x P 2 Q 2. (1) The above equatio tells us that we ca divide the problem ito 4 subproblems with half of the origial iput size, which gives us the followig recurrece relatio: T () = 4T ( 2 ) + O() However, we ca improve this to T () = 3T ( 2 ) + O() by the fact that P 1 Q 2 + P 2 Q 1 = (P 1 + P 2 )(Q 1 + Q 2 ) P 1 Q 1 P 2 Q 2. (2) Note that P 1 Q 1 ad P 2 Q 2 i equatio 2 also appear i equatio 1. Therefore we oly eed to compute oe more multiplicatio, which is (P 1 + P 2 )(Q 1 + Q 2 ), givig us the followig recurrece relatio T () = 3T ( 2 ) + O() This recurrece relatio gives us T () = O( log 2 3 ), which is faster tha aively multiplyig out all terms. 7 Media Fidig ad Selectio Suppose we are give a elemet array A ad we wish to fid its media. A immediate algorithm to this problem would be to first sort A usig your favorite O(log)-time sortig algorithm. The, we ca look at (/2)th elemet i this sorted array to fid the media. However, this approach seems to beig too much work the added time it takes to order all the elemets is uecessary sice we re oly iterested i fidig the media. Ideally, we could fid the media i O() time (a algorithm that solves this problem must ru i Ω() time; ca you argue why?) Istead of fidig the media of A, we will actually solve the problem of selectio istead, which is a slightly more geeral problem. Specifically, the algorithm is ow give a parameter k as iput ad is asked to fid the kth smallest item i A (i.e., the elemet we would fid at idex k if A was sorted). Clearly, solvig selectio solves the problem of media fidig if we set k = /2. This more geeral structure will allow us to correctly defie our coquer step whe devisig our divide-ad-coquer algorithm. 2-6
7.1 A DAC Algorithm for Selectio Our goal is to defie a algorithm that fids the kth smallest elemet of a elemet A i O() time. We begi by defiig the followig correct (but as we will see) iefficiet algorithm SELECTION(A,k): 1. Select a pivot value p A (for ow, we do this arbitrarily). The, restructure A so that it is the cocateatio of the followig three subarrays: all elemets less tha p, followed by elemets equal to p, ad the followed by elemets greater tha p; call these regios A 1, A 2, ad A 3, respectively. We will also deote that A i = i. 2. If k 1, we retur SELECTION(A 1,k). Sice A 1 cotais the 1 smallest elemets i A ad k 1, we kow that the kth smallest elemet i A 1 is also the kth smallest elemet i A. Thus makig a recursive call o A 1 that leaves the value of k uchaged will correctly retur the desired elemet. 3. If 1 < k 1 + 2, simply retur p. Here we kow that the kth smallest elemet lies i A 2. Sice this subarray oly cotais the value p, the desired etry must be the pivot p. 4. Otherwise, k > 1 + 2, i which case we retur SELECTION(A 3,k 1 2 ). Now we kow our desired elemet is i subarray A 3 ; however, because this array is preceded by A 1 ad A 2, we ow must locate the (k 1 2 )th smallest elemet i A 3 if we wat to fid the kth smallest elemet i the overall array A (this just accouts for the precedig elemets that we are essetially choppig-off whe we make a recursive call that oly examies A 3 ). This completes the algorithm s defiitio (ote that here, step 1 defies the divide step, ad steps 2-4 defie our coquer step; there is o combie procedure i this case). It should ow be clear that this algorithm is ideed correct; however, if we are ot careful as to how we pick our pivot p, the ruig time of the algorithm will suffer i the same way we saw quick-sort suffer i the last lecture. More specifically, let T () be the ruig-time of the algorithm o a elemet array. Recall that step 1 (i.e., pickig a pivot ad restructurig the array) ca be doe i O() time. The ruig time of steps 2-4 will deped o what case we fall i, but regardless, we ca say it is bouded by T (max{ 1, 3 }) sice we oly ever make recursive calls o A 1 or A 3 (but ot both). If we get a eve split each time we make a recursive call ad 1 3 /2, we will be i good shape. Formally, the ruig time of the algorithm i this case is give by: T () = T (/2) + O() c ( + /2 + /4 +... + 1) (3) = c (1 + 1/2 + 1/4 +... + 1/) (4) < c 2 = O(), (5) where c is the costat hidde by the O() term i the origial recurrece. It should be fairly straightforward to see why if we expad the recurrece, we obtai lie (3). To see why iequality (5) is true, we quickly review geometric series. Recall (hopefully) that for some real umber 0 r < 1, we have that 1 + r + r 2 + r 3 +... = i=0 r i = 1 1 r. (6) For lie (4), we have that r = 1/2 whe examiig the term (1 + 1/2 + 1/4 +... + 1/). Sice this sum is bouded by i=0 (1/2)i (the latter is a ifiite sum whose terms subsumes the fiite series i the former), 2-7
the closed form for a ifiite geometric series give by (6) implies that these terms are bouded by 1/(1 1/2) = 2. Turig our attetio back to SELECTION, the case where A 1 ad A 3 are roughly equal i size will result i this ideal ruig time; however, sice we curretly have o rule for pickig a pivot, there is othig prevetig, say, 1 = 0 ad 3 = 1. If this worst case occurs for every recursive call we make, the ruig time is ow: T () = T ( 1) + O() = c ( + 1 + 2 +... + 1) = O( 2 ). Thus, as curretly defied, the ruig time of SELECTION(A,k) is O( 2 ), which is worse tha the O(log) aive sortig approach we first gave. Clearly, if we wat to get ay pay dirt out of this algorithm, we will have to fid a way of pickig the pivot so that we maitai at least some balace betwee the sizes of A 1 ad A 3. 7.2 Pickig a Good Pivot As outlied above, we would ideally like to pick a pivot so that 1 3 /2. The best pivot to pick the is just the media of A; however, this is a bit circular. Selectio is a problem that, at least whe k = /2, is tryig to fid the media i the first place! Therefore, the somewhat arcae pivot-fidig procedure we are about to defie attempts to fid a approximate media i O() time. If there is at least some balace betwee A 1 ad A 3, hopefully we ca discard a large eough fractio of the array i each recursive call to yield our desired O() rutime. Our ew pivot procedure, which we will call PIVOT-FIND(A), is defied as follows (where agai A = ): 1. Divide A ito /5 blocks B 1,...,B /5 each of size 5 (assume for sake simplicity that is divisible by 5). 2. Sort each of these blocks idividually, ad so afterwards, the 5 elemets i a give block B i are sorted with respect to oe aother. Note that the media of each block is ow the third elemet withi that block. 3. For each block B i, store B i s media i a ew array C of size /5. 4. We the retur our pivot to be SELECTION(C,/10), which is the media of C (/10 comes from the fact there are /5 elemets i C, ad therefore the midway poit i this array will be at /10). The most iterestig observatio to make about this procedure is the fact we are usig a recursive call to SELECTION as our meas of pickig the pivot. Thus, we are couterituitively usig the very dividead-coquer algorithm we are attemptig to defie as a subroutie whe defiig our divide step (pickig a pivot). So for example, cosider our first recursive call to SELECTION(A,k). By the time PIVOT-FIND(A) completes, we will have made several cascadig calls to SELECTION ad PIVOT-FIND o smaller arrays before eve reachig the first recursive coquer steps i our top recursive call (steps 2-4 i SELECTION). By o meas is this a typical approach whe defiig divide-ad-coquer algorithms, but it is a great example of how oe ca take a algorithmic paradigm ad creatively dovetail stadard techiques i order to costruct a faster algorithm. 2-8
Also ote that this procedure ideed rus i O() time. Step 1-2 will take O() time sice there are /5 blocks, each of which take O(1) time to sort sice they each have a costat umber of elemets (eve if we use bubble-sort or isertio-sort). Steps 3-4 will take O() time, as well, sice we ca step through all the blocks i /5 steps, pluck out each media at a give block s third locatio, ad the add it to C; hece, the procedure s overall ruig time is i O(). Now, let s see what this ew PIVOT-FIND(A) buys us. To do our aalysis, cosider reorderig the blocks so that they are sorted by their medias. More formally, we specify this orderig as B φ(1), B φ(2),...,b φ(/5), where φ is a fuctio that remaps the block idices such that if m i is the media of block B i, we have that m φ(i) m φ( j) for all j < i ad m φ(i) m φ( j) for all j > i. It is importat to ote that the algorithm is ot actually performig this reorderig we are just defiig this structure for sake of aalysis. Observe that the pivot p retured by SELECTION(C,/10) is the media of block B φ(/10). Now, defie the followig sets: Let S 1 be the set of all x A such that x B φ( j), x m φ( j), ad j /10 (recall that we defied m i to be the media of block B i ). Iformally, these are elemets that exist i the first half of this block orderig ad are less or equal to the media of the block to which they belog. Similarly, let S 2 be the set of y A such that y B φ( j), y m φ( j), ad j /10. Likewise, this is the set of elemets that exist i the secod half of our block orderig ad are o smaller tha the medias of their respective blocks. Figure 1 illustrates the defiitios of these sets. So what is all this structure good for? Remember our goal is to make sure that both A 1 ad A 3 receive a large eough fractio of the elemets. The followig lemma makes use of our block orderig B φ(1),...,b φ(/5) ad our ewly defied sets S 1 ad S 2 to argue that if we use PIVOT-FIND to fid our pivot, we ideed obtai some balace. Lemma 1. If we use PIVOT-FIND to pick the pivot p, the max{ 1, 3 } 7/10. Proof. Our key observatios are that S 1 A 1 A 2 ad S 2 A 2 A 3. Here, we will argue that the former claim is true ad show it implies that 3 7/10. We will leave establishig S 2 A 2 A 3 ad showig that this implies 1 7/10 as a exercise (although, it should be symmetric to the argumet we preset here). Let x S 1. Therefore, x exists i some block B φ( j) where j /10 ad is less tha or equal to the media of its block m φ( j). Sice j /10, m φ( j) must lie i the first half of C ad therefore is o greater tha the media of C. Sice PIVOT-FIND esures that p is the media of C, we have that m φ( j) p. Thus, it follows that x p, implyig that x A 1 A 2, as desired. To complete the proof, observe that S 1 cotais 3/5 of the elemets i blocks B φ(1),...,b φ(/10), sice for each of these blocks, S 1 icludes the media ad the two precedig elemets. Sice these blocks accout for half the elemets i etire array A, it follows S 1 = 3/10. However, we just showed S 1 is a subset of A 1 A 2, implyig that A 1 A 2 caot be smaller tha 3/10. This implies that A 3 = 3 ca be o larger tha 7/10 sice 1 + 2 + 3 =. As metioed at the start of the proof, a symmetric argumet ca be made to show 1 7/10. Hece, we have that max{ 1, 3 } 7/10. We are ow ready to complete our ru-time aalysis for SELECTION with PIVOT-FIND. Let s briefly recall the ruig times of all the compoets i a give recursive call of SELECTION(A, k), where agai we deote T () to be the total ruig time of this call if A =. 2-9
B (1) B (2). B (5). B (9) S 1 C S 2 10 14 15 23 40 12 13 17 83 91 20 21 22 30 31 29 31 35 91 99 45 49 50 51 69 11 12 64 67 80 72 73 75 88 89 22 23 81 83 99 10 11 97 98 99 Figure 1: A diagram illustratig the defiitios of B φ(1),...,b φ(/5), C, S 1, ad S 2. Here = 45, ad thus we have 9 blocks i total. Observe that each block is sorted idividually ad that the blocks themselves are ordered based o their medias. Array C is outlied i red, ad the media of C, circled i pik, would serve as our pivot p. Sets S 1 ad S 2 are highlighted by the blue ad gree boxes, respectively. Also observe that all the elemets i S 1 ad S 2 are less tha ad greater tha the pivot, respectively (which is argued formally i Lemma 1). 2-10
Steps 1-3 of PIVOT-FIND, where we divide our array ito blocks ad create the array of medias C, takes O() time. Step 4 of PIVOT-FIND makes a call to SELECTION(C,/10); sice C is a array of size /5, this will take T (/5) time. Restructurig the array aroud the pivot i steps 1-2 of SELECTION takes O() time. As we argued earlier, steps 2-4 of SELECTION take T (max{ 1, 3 }) time. By Lemma 1, we kow that this will be at most T (7/10). Thus, the overall ruig time of the algorithm is as follows (we will agai use c as the costat that is hidde by the O() term that bouds the ruig times of creatig array C ad pivotig the array aroud p). T () = T (7/10) + T (/5) + c ( ) 9 i < c (7) 10 i=0 = c 10 = O(), (8) as desired. You should try to verify iequality (7) (use the tree expasio method; you should get that the total work doe o the kth level of the tree is c (9/10) k ). Equatio (8) follows by the closed form for a geometric series we saw earlier i equatio (6). A fial ote: observe that if we had picked the size of each block to be 4 istead of 5, the recurrece would ow become T () = T (3/4) + T (/4) + O(). Whe we expad the recursio tree for this recurrece, we are doig c work at each level of the tree. Sice the tree will have O(log) levels, we istead get a ruig time of O(log). Thus, the decisio to use blocks of 5 was carefully made whe desigig the algorithm to avoid gettig this extra O(log) factor. Pickig aythig greater tha 5 will also work, but we will still get a O() time algorithm sice doig this will oly improve the costat we get o lie (8) (ad remember, ay algorithm for selectio must take Ω() time). 8 Master Theorem Let us aalyze a recurrece relatio for a geeral divide-ad-coquer algorithm: T () = at ( b ) + O(d ). The computatio tree is like: O( d ) a braches /b O(a(/b) d ) O(a k (/b k ) d ) log b levels 2-11
We have T () = O( d +a( b )d +...+a k ( ) d +...+a log b k b ( ) d ), where the item a log b log b b ( ) d = a log b log b b = log b a. The solutio is based o the compariso betwee d ad log b a. There are three situatios: d > log b a: T () = log b 1 j=0 a j ( ) d = d a b j b d log b a 1 a d < log b a: T () = log b 1 j=0 a j ( ) d = d a b j O( log b a ). b d. The item d domiates ad we have T () = O( d ). b d log b a 1 a b d. The item log b a domiates ad we have T () = d = log b a: All items have the same power ad we have T () = O( log b 1 j=0 a j b d j d ) = O( d log b ). 2-12