Algorithms. Algorithms. Algorithms 2.2 M ERGESORT. mergesort bottom-up mergesort. sorting complexity divide-and-conquer

Transcription

1 Algorthms Two classc sortng algorthms: mergesort and qucsort R OBERT S EDGEWICK K EVIN W AYNE Crtcal components n the world s computatonal nfrastructure. Full scentfc understandng of ther propertes has enabled us to develop them nto practcal system sorts. Qucsort honored as one of top 10 algorthms of 20 th 2.2 M ERGESORT mergesort bottom-up mergesort Algorthms F O U R T H E D I T I O N century n scence and engneerng. Mergesort. [ths lecture]... sortng complexty dvde-and-conquer Qucsort. [next lecture] R OBERT S EDGEWICK K EVIN W AYNE Last updated on 2/24/16 8:14 PM 2 PSA 2.2 M ERGESORT Mae sure to regster your Clcer on blacboard mergesort bottom-up mergesort You can mss up to 3 lectures wth no penalty wthout any vald reason After that, emal Maa wth documentaton of why you couldn t attend Algorthms R OBERT S EDGEWICK K EVIN W AYNE 3 sortng complexty dvde-and-conquer

2 Mergesort Mergng demo Basc plan. Dvde array nto two halves. Recursvely sort each half. Merge two halves. nput sort left half sort rght half merge results M E R G E S O R T E X A M P L E E E G M O R R S T E X A M P L E E E G M O R R S A E E L M P T X A E E E E G L M M O P R R S T X lo md md+1 h Mergesort overvew sorted sorted 5 6 Mergng demo Mergng demo lo md md+1 h copy to auxlary array

3 Mergng demo Mergng demo A compare mnmum n each subarray compare mnmum n each subarray Mergng demo Mergng demo A E G M R A C E R T A EC G M R A C E R T compare mnmum n each subarray compare mnmum n each subarray

4 Mergng demo Mergng demo A C G M R A C E R T A C GE M R A C E R T compare mnmum n each subarray compare mnmum n each subarray Mergng demo Mergng demo A C E M R A C E R T A C E ME R A C E R T compare mnmum n each subarray compare mnmum n each subarray

5 Mergng demo Mergng demo A C E E R A C E R T A C E E RE A C E R T compare mnmum n each subarray compare mnmum n each subarray A C E R T Mergng demo Mergng demo A C E E E A C E R T A C E E E AG C E R T compare mnmum n each subarray compare mnmum n each subarray

6 Mergng demo Mergng demo A C E E E G C E R T A C E E E G MC E R T compare mnmum n each subarray compare mnmum n each subarray Mergng demo Mergng demo A C E E E G M E R T A C E E E G M ER R T compare mnmum n each subarray compare mnmum n each subarray

7 Mergng demo Mergng demo A C E R T A C E R T one subarray exhausted, tae from other one subarray exhausted, tae from other Mergng demo Mergng demo A C E R T A C E R T one subarray exhausted, tae from other one subarray exhausted, tae from other

8 Mergng demo Mergng demo lo h A C E R T A C E R T both subarrays exhausted, done sorted Mergng: Java mplementaton Mergesort quz 1 prvate statc vod merge(comparable[] a, Comparable[] aux, nt lo, nt md, nt h) for (nt = lo; <= h; ++) aux[] = a[]; nt = lo, = md+1; for (nt = lo; <= h; ++) f ( > md) a[] = aux[++]; else f ( > h) a[] = aux[++]; else f (less(aux[], aux[])) a[] = aux[++]; else a[] = aux[++]; copy merge How many calls does merge() mae to to less() to merge two sorted subarrays of sze N / 2 each nto a sorted array of sze N. A. ~ ¼ N to ~ ½ N B. ~ ½ N best-case nput (N/2 compares) C. ~ ½ N to ~ N A B C D E F G H D. ~ N E. Hey, ths ust counts for class partcpaton ponts, rght? lo md h A G L O R H I M S T A G H I L M worst-case nput (N - 1 compares) A B C H D E F G Q. Why s aux passed as argument? Why s md passed as argument? 31 32

9 Mergesort: Java mplementaton Mergesort: trace publc class Merge prvate statc vod merge(...) /* as before */ prvate statc vod sort(comparable[] a, Comparable[] aux, nt lo, nt h) f (h <= lo) return; nt md = lo + (h - lo) / 2; sort(a, aux, lo, md); sort(a, aux, md+1, h); merge(a, aux, lo, md, h); publc statc vod sort(comparable[] a) Comparable[] aux = new Comparable[a.length]; sort(a, aux, 0, a.length - 1); lo h merge(a, aux, 0, 0, 1) merge(a, aux, 2, 2, 3) merge(a, aux, 0, 1, 3) merge(a, aux, 4, 4, 5) merge(a, aux, 6, 6, 7) merge(a, aux, 4, 5, 7) merge(a, aux, 0, 3, 7) merge(a, aux, 8, 8, 9) merge(a, aux, 10, 10, 11) merge(a, aux, 8, 9, 11) merge(a, aux, 12, 12, 13) merge(a, aux, 14, 14, 15) merge(a, aux, 12, 13, 15) merge(a, aux, 8, 11, 15) merge(a, aux, 0, 7, 15) M E R G E S O R T E X A M P L E E M R G E S O R T E X A M P L E E M G R E S O R T E X A M P L E E G M R E S O R T E X A M P L E E G M R E S O R T E X A M P L E E G M R E S O R T E X A M P L E E G M R E O R S T E X A M P L E E E G M O R R S T E X A M P L E E E G M O R R S E T X A M P L E E E G M O R R S E T A X M P L E E E G M O R R S A E T X M P L E E E G M O R R S A E T X M P L E E E G M O R R S A E T X M P E L E E G M O R R S A E T X E L M P E E G M O R R S A E E L M P T X A E E E E G L M M O P R R S T X Trace of merge results for top-down mergesort lo md h result after recursve call Mergesort quz 2 Mergesort: anmaton Whch of the followng subarray lengths wll occur when runnng mergesort on an array of length 12? 50 random tems A. 1, 2, 3, 4, 6, 8, 12 B. 1, 2, 3, 6, 12 C. 1, 2, 4, 8, 12 D. 1, 3, 6, 9, 12 E. I don't now algorthm poston n order current subarray not n order 35 36

10 Mergesort: anmaton Mergesort analyss: number of compares 50 reverse-sorted tems Proposton. Mergesort uses N lg N compares to sort an array of length N. Pf setch. The maxmum number of compares C (N) to mergesort an array of length N satsfes the recurrence: algorthm poston n order current subarray not n order C (N) C ( N / 2 ) + C ( N / 2 ) + N 1 for N > 1, wth C (1) = 0. left half rght half merge We solve ths smpler recurrence, and assume N s a power of 2: D (N) = 2 D (N / 2) + N, for N > 1, wth D (1) = 0. Q. Can you show that C (N) C(N+1)? result holds for all N (analyss cleaner n ths case) Dvde-and-conquer recurrence Proposton. If D (N) satsfes D (N) = 2 D (N / 2) + N for N > 1, wth D (1) = 0, then D (N) = N lg N. Pf by pcture. [assumng N s a power of 2] Mergesort analyss: number of array accesses Proposton. Mergesort uses 6 N lg N array accesses to sort an array of length N. Pf setch. The max number of array accesses A (N) satsfes the recurrence: D (N) N = N A (N) A ( N / 2 ) + A ( N / 2 ) + 6 N for N > 1, wth A (1) = 0. D (N / 2) D (N / 2) 2 (N/2) = N Key pont. Any algorthm wth the followng structure taes N log N tme: lg N D(N / 4) D(N / 4) D(N / 4) D(N / 4) D(N / 8) D(N / 8) D(N / 8) D(N / 8) D(N / 8) D(N / 8) D(N / 8) D(N / 8) 4 (N/4) = N 8 (N/8) = N T(N) = N lg N publc statc vod f(nt N) f (N == 0) return; f(n/2); f(n/2); lnear(n); solve two problems of half the sze do a lnear amount of wor Notable examples. FFT, hdden-lne removal, Kendall-tau dstance, 39 40

11 Mergesort analyss: memory Mergng demo Proposton. Mergesort uses extra space proportonal to N. Pf. The array needs to be of length N for the last merge. two sorted subarrays A C D G H I M N U V B E F J O P Q R S T A B C D E F G H I J M N O P Q R S T U V merged result Def. A sortng algorthm s n-place f t uses c log N extra memory. Ex. Inserton sort, selecton sort, shellsort. Challenge 1 (not hard). Use array of length ~ ½ N nstead of N. Challenge 2 (very hard). In-place merge. [Kronrod 1969] lo md md+1 h sorted sorted 41 Mergng demo Mergng demo lo md md+1 h copy to auxlary array (of half the sze)

12 Mergng demo Mergng demo A compare mnmum n each subarray compare mnmum n each subarray Mergng demo Mergng demo A E G M R A C E R T A EC G M R A C E R T compare mnmum n each subarray compare mnmum n each subarray

13 Mergng demo Mergng demo A C G M R A C E R T A C GE M R A C E R T compare mnmum n each subarray compare mnmum n each subarray Mergng demo Mergng demo A C E M R A C E R T A C E ME R A C E R T compare mnmum n each subarray compare mnmum n each subarray

14 Mergng demo Mergng demo A C E E R A C E R T A C E E RE A C E R T compare mnmum n each subarray compare mnmum n each subarray Mergng demo Mergng demo A C E E E A C E R T A C E E E AG C E R T compare mnmum n each subarray compare mnmum n each subarray

15 Mergng demo Mergng demo A C E E E G C E R T A C E E E G CM E R T compare mnmum n each subarray compare mnmum n each subarray Mergng demo Mergng demo A C E E E G M E R T A C E E E G M ER R T compare mnmum n each subarray compare mnmum n each subarray

16 Mergng demo Mergng demo lo h A C E R T A C E R T f auxlary subarray s exhausted, done! sorted Mergesort quz 3 Stablty: mergesort Is our mplementaton of mergesort stable? A. Yes. B. No, but t can be modfed to be stable. C. No, mergesort s nherently unstable. D. I don't remember what stablty means. E. I don't now. a sortng algorthm s stable f t preserves the relatve order of equal eys nput C A1 B A2 A3 sorted A3 A1 A2 B C not stable Proposton. Mergesort s stable. publc class Merge prvate statc vod merge(...) /* as before */ prvate statc vod sort(comparable[] a, Comparable[] aux, nt lo, nt h) f (h <= lo) return; nt md = lo + (h - lo) / 2; sort(a, aux, lo, md); sort(a, aux, md+1, h); merge(a, aux, lo, md, h); publc statc vod sort(comparable[] a) /* as before */ Pf. Suffces to verfy that merge operaton s stable

17 Stablty: mergesort Mergesort: practcal mprovements Proposton. Merge operaton s stable. prvate statc vod merge(...) for (nt = lo; <= h; ++) aux[] = a[]; nt = lo, = md+1; for (nt = lo; <= h; ++) f ( > md) a[] = aux[++]; else f ( > h) a[] = aux[++]; else f (less(aux[], aux[])) a[] = aux[++]; else a[] = aux[++]; A1 A2 A3 B D Pf. Taes from left subarray f equal eys A4 A5 C E F G Use nserton sort for small subarrays. Mergesort has too much overhead for tny subarrays. Not captured n cost model (number of compares) Cutoff to nserton sort for 10 tems. prvate statc vod sort(comparable[] a, Comparable[] aux, nt lo, nt h) f (h <= lo + CUTOFF - 1) Inserton.sort(a, lo, h); return; nt md = lo + (h - lo) / 2; sort (a, aux, lo, md); sort (a, aux, md+1, h); merge(a, aux, lo, md, h); Mergesort wth cutoff to nserton sort: vsualzaton Mergesort: practcal mprovements frst subarray second subarray frst merge Stop f already sorted. Is largest tem n frst half smallest tem n second half? Helps for partally-ordered arrays. A B C D E F G H I J A B C D E F G H I J M N O P Q R S T U V M N O P Q R S T U V frst half sorted second half sorted prvate statc vod sort(comparable[] a, Comparable[] aux, nt lo, nt h) f (h <= lo) return; nt md = lo + (h - lo) / 2; sort (a, aux, lo, md); sort (a, aux, md+1, h); f (!less(a[md+1], a[md])) return; merge(a, aux, lo, md, h); result 67 68

18 Mergesort: practcal mprovements Java 6 system sort Elmnate the copy to the auxlary array. Save tme (but not space) by swtchng the role of the nput and auxlary array n each recursve call. prvate statc vod merge(comparable[] a, Comparable[] aux, nt lo, nt md, nt h) nt = lo, = md+1; for (nt = lo; <= h; ++) f ( > md) aux[] = a[++]; else f ( > h) aux[] = a[++]; else f (less(a[], a[])) aux[] = a[++]; else aux[] = a[++]; prvate statc vod sort(comparable[] a, Comparable[] aux, nt lo, nt h) f (h <= lo) return; nt md = lo + (h - lo) / 2; sort (aux, a, lo, md); sort (aux, a, md+1, h); merge(a, aux, lo, md, h); merge from to assumes s ntalze to once, before recursve calls Basc algorthm for sortng obects = mergesort. Cutoff to nserton sort = 7. Stop-f-already-sorted test. Elmnate-the-copy-to-the-auxlary-array trc. Arrays.sort(a) swtch roles of and Bottom-up mergesort Basc plan. Pass through array, mergng subarrays of sze 1. Repeat for subarrays of sze 2, 4, 8,... Algorthms ROBERT SEDGEWICK KEVIN WAYNE MERGESORT mergesort bottom-up mergesort sortng complexty dvde-and-conquer sz = 1 merge(a, aux, 0, 0, 1) merge(a, aux, 2, 2, 3) merge(a, aux, 4, 4, 5) merge(a, aux, 6, 6, 7) merge(a, aux, 8, 8, 9) merge(a, aux, 10, 10, 11) merge(a, aux, 12, 12, 13) merge(a, aux, 14, 14, 15) sz = 2 merge(a, aux, 0, 1, 3) merge(a, aux, 4, 5, 7) merge(a, aux, 8, 9, 11) merge(a, aux, 12, 13, 15) sz = 4 merge(a, aux, 0, 3, 7) merge(a, aux, 8, 11, 15) sz = 8 merge(a, aux, 0, 7, 15) a[] M E R G E S O R T E X A M P L E E M R G E S O R T E X A M P L E E M G R E S O R T E X A M P L E E M G R E S O R T E X A M P L E E M G R E S O R T E X A M P L E E M G R E S O R E T X A M P L E E M G R E S O R E T A X M P L E E M G R E S O R E T A X M P L E E M G R E S O R E T A X M P E L E G M R E S O R E T A X M P E L E G M R E O R S E T A X M P E L E G M R E O R S A E T X M P E L E G M R E O R S A E T X E L M P E E G M O R R S A E T X E L M P E E G M O R R S A E E L M P T X A E E E E G L M M O P R R S T X 72

19 Bottom-up mergesort: Java mplementaton Mergesort: vsualzatons publc class MergeBU prvate statc vod merge(...) /* as before */ publc statc vod sort(comparable[] a) nt N = a.length; Comparable[] aux = new Comparable[N]; for (nt sz = 1; sz < N; sz = sz+sz) for (nt lo = 0; lo < N-sz; lo += sz+sz) merge(a, aux, lo, lo+sz-1, Math.mn(lo+sz+sz-1, N-1)); Bottom lne. Smple and non-recursve verson of mergesort. 73 top-down mergesort (cutoff = 12) bottom-up mergesort (cutoff = 12) 74 Mergesort quz 4 Natural mergesort Whch s faster n practce: top-down mergesort or bottom-up mergesort? You may assume N s a power of 2. A. Top-down (recursve) mergesort. Maybe! Localty B. Bottom-up (nonrecursve) mergesort. Maybe! Overhead C. About the same. D. It depends. E. I don't now. Overhead can be mnmzed wth well-chosen cutoff to nserton sort. Localty s nherent. 75 Idea. Explot pre-exstng order by dentfyng naturally-occurrng runs. nput frst run second run merge two runs Tradeoff. Fewer passes vs. extra compares per pass to dentfy runs. 76

20 Tmsort Natural mergesort. Use bnary nserton sort to mae ntal runs (f needed). A few more clever optmzatons. Consequence. Lnear tme on many arrays wth pre-exstng order. Now wdely used. Python, Java 7, GNU Octave, Androd,. Tm Peters Sortng summary nplace? stable? best average worst remars selecton ½ N 2 ½ N 2 ½ N 2 N exchanges nserton N ¼ N 2 ½ N 2 use for small N or partally ordered shell N log3 N? c N 3/2 tght code; subquadratc merge ½ N lg N N lg N N lg N tmsort N N lg N N lg N N log N guarantee; stable mproves mergesort when preexstng order? N N lg N N lg N holy sortng gral Commercal brea 2.2 MERGESORT Algorthms mergesort bottom-up mergesort sortng complexty dvde-and-conquer ROBERT SEDGEWICK KEVIN WAYNE

21 Complexty of sortng Decson tree (for 3 dstnct eys a, b, and c) Computatonal complexty. Framewor to study effcency of algorthms for solvng a partcular problem X. Model of computaton. Allowable operatons. Cost model. Operaton counts. Upper bound. Cost guarantee provded by some algorthm for X. Lower bound. Proven lmt on cost guarantee of all algorthms for X. Optmal algorthm. Algorthm wth best possble cost guarantee for X. yes b < c a < b yes no code between compares (e.g., sequence of exchanges) no yes a < c no heght of tree = worst-case number of compares model of computaton cost model decson tree # compares lower bound ~ upper bound can access nformaton only through compares (e.g., Java Comparable framewor) a b c yes a < c no b a c yes b < c no upper bound ~ N lg N from mergesort lower bound? a c b c a b b c a c b a optmal algorthm? each leaf corresponds to one (and only one) orderng; (at least) one leaf for each possble orderng complexty of sortng Compare-based lower bound for sortng Proposton. Any compare-based sortng algorthm must use at least lg ( N! ) ~ N lg N compares n the worst-case. Pf. Assume array conssts of N dstnct values a1 through an. Worst case dctated by heght h of decson tree. Bnary tree of heght h has at most 2h leaves. N! dfferent orderngs at least N! leaves. Compare-based lower bound for sortng Proposton. Any compare-based sortng algorthm must use at least lg ( N! ) ~ N lg N compares n the worst-case. Pf. Assume array conssts of N dstnct values a1 through an. Worst case dctated by heght h of decson tree. Bnary tree of heght h has at most 2h leaves. N! dfferent orderngs at least N! leaves. h 2 h # leaves N! h lg ( N! ) ~ N lg N Strlng's formula at least N! leaves no more than 2 h leaves 83 84

22 Complexty of sortng Complexty results n context Model of computaton. Allowable operatons. Cost model. Operaton count(s). Upper bound. Cost guarantee provded by some algorthm for X. Lower bound. Proven lmt on cost guarantee of all algorthms for X. Optmal algorthm. Algorthm wth best possble cost guarantee for X. model of computaton decson tree cost model # compares upper bound ~ N lg N lower bound ~ N lg N optmal algorthm mergesort complexty of sortng Frst goal of algorthm desgn: optmal algorthms. Compares? Mergesort s optmal wth respect to number compares. Space? Mergesort s not optmal wth respect to space usage. Lessons. Use theory as a gude. Ex. Desgn sortng algorthm that guarantees ~ ½ N lg N compares? Ex. Desgn sortng algorthm that s both tme- and space-optmal? Complexty results n context (contnued) Commonly-used notatons n the theory of algorthms Lower bound may not hold f the algorthm can tae advantage of: The ntal order of the nput. Ex: nserton sort requres only a lnear number of compares on partally-sorted arrays. notaton provdes example shorthand for Tlde leadng term ~ ½ N 2 ½ N 2 ½ N N log N + 3 N The dstrbuton of ey values. Ex: 3-way qucsort requres only a lnear number of compares on arrays wth a constant number of dstnct eys. [stay tuned] The representaton of the eys. Ex: radx sorts requre no ey compares they access the data va character/dgt compares. Q. How would you sort an array of Students by brthday? Q. How would you sort an array of Students by last name (of <= 12 chars)? Bg Theta order of growth Θ(N 2 ) Bg O upper bound O(N 2 ) Bg Omega lower bound Ω(N 2 ) ½ N 2 10 N 2 5 N N log N + 3 N 10 N N 22 N log N + 3 N ½ N 2 N 5 N N log N + 3 N 87 88

23 Shuffle a lned lst 2.2 MERGESORT Problem. Gven a sngly-lned lst, rearrange ts nodes unformly at random. Assumpton. Access to a perfect random number generator. all N! permutatons equally lely Verson 1. Lnear tme, lnear extra space. Verson 2. Lnearthmc tme, logarthmc or constant extra space. mergesort Hard! (See Pazza) bottom-up mergesort frst Algorthms sortng complexty dvde-and-conquer nput null ROBERT SEDGEWICK KEVIN WAYNE frst shuffled null 90