Tony Wong 2017-03-04
Sorting Sorting is the reordering of array elements into a specific order (usually ascending) For example, array a[0..8] consists of the final exam scores of the n = 9 students in a class. After sorting, a[0] contains the lowest score and a[8] contains the highest score. Original a[0] a[1] a[2] a[3] a[4] a[5] a[6] a[7] a[8] Value 12 8 4 5 4 7 4 20 3 Sorted a[0] a[1] a[2] a[3] a[4] a[5] a[6] a[7] a[8] Value 3 4 4 4 5 7 8 12 20 2
Sorting "What is the most efficient way to sort 1 million 32-bit integers?" 3
Sorting Algorithms There are many ways to sort an array, just like we all use different ways to sort a deck of playing cards Some are faster, and some are slower Some are easier to code, some are harder to code Comparison Based Sorting (Slow) Selection sort Bubble sort Insertion sort Comparison Based Sorting (Fast) Shell Sort Quick Sort Merge Sort Non-comparison Based Sorting Counting Sort Radix Sort 4
Selection Sort ( 選擇排序法 ) Selection sort works by repeatedly moving the maximum element in the unsorted part to the front of the sorted part Imagine that you are playing 鋤大 D and is given 13 playing cards, how would you sort them? Find the 2(s) and place it/them at the back Find the Ace(s) and place it/them at the back (before the 2s) and so on 5
Selection Sort Let s count i from n 1 down to 1 In the each round, the greatest element needs to go to a[i] Instead of moving, swap the elements a[maxj] and a[i] Round a[0] a[1] a[2] a[3] a[4] a[5] i = 5 2 4 6 1 3 5 i = 4 2 4 5 1 3 6 i = 3 2 4 3 1 5 6 i = 2 2 1 3 4 5 6 i = 1 2 1 3 4 5 6 Result 1 2 3 4 5 6 maxj=2 maxj=2 maxj=1 maxj=2 maxj=0 7
Selection Sort Find the max element in a[0].. a[i] Swapping the two numbers (place the maximum at a[i]) 8
Selection Sort Alternatively, count i = 0.. n - 2 and move the minimum element to a[i] Round a[0] a[1] a[2] a[3] a[4] a[5] i = 0 2 4 6 1 3 5 i = 1 1 4 6 2 3 5 i = 2 1 2 6 4 3 5 i = 3 1 2 3 4 6 5 i = 4 1 2 3 4 6 5 Result 1 2 3 4 5 6 9
Selection Sort Analysis Q: What is the number of comparisons performed? Q: What is the number of swaps performed? 10
Properties of Selection Sort Task: It is the algorithm for sorting an array using M1602 Valentine Roses minimum number of pairwise swaps if we do not swap elements that are already in the correct position Max number of swaps required: n 1 e.g. sorting (2, 3, 4, 5, 6, 1) requires 5 pairwise swaps Time complexity: Worst-case = Best-case = Average-case = Number of comparisons performed = n + n 1 + n 2 + + 1 = n 1 (n) = O n 2 2 Space complexity: O(1) the only extra variables are i, j, maxj/minj and t 11
Bubble Sort ( 冒泡排序法 ) There is a class of n students randomly placed in a straight line. Each student can only see the students before and after him Sort the students according to class number Starting from the beginning of the array, compare two adjacent elements: If the left one is greater than the right one, that is, a[i] > a[i + 1], swap them After we process the array for one round ("pass"), the greatest element will be in the correct place (right-most). Repeat this process for a total of n-1 passes 12
Bubble Sort Loop i = n 1.. 1 Inside the i for-loop, loop j = 0.. j-1 a[j] and a[j + 1] are compared if a[j] > a[j + 1] swap a[j] and a[j + 1] Round a[0] a[1] a[2] a[3] a[4] i = 4, j = 0 2 4 5 1 3 i = 4, j = 1 2 4 5 1 3 i = 4, j = 2 2 4 5 1 3 i = 4, j = 3 2 4 1 5 3 i = 3, j = 0 2 4 1 3 5 i = 3, j = 1 2 4 1 3 5 i = 3, j = 2 2 1 4 3 5 i = 2, j = 0 2 1 3 4 5 i = 2, j = 1 1 2 3 4 5 i = 1, j = 0 1 2 3 4 5 Result 1 2 3 4 5 13
Bubble Sort If the left one is greater than the right one, Swap the two numbers 14
Bubble Sort Analysis Q: What is the number of comparisons performed? Q: What is the actual number of swaps performed? 15
Properties of Bubble Sort It is the algorithm for sorting an array using minimum number of adjacent pairwise swaps n 1 (n) Max number of swaps required: 2 e.g. sorting (6, 5, 4, 3, 2, 1) requires 15 pairwise swaps Time complexity: Worst-case = Best-case = Average-case = Number of comparisons performed = n + n 1 + n 2 + + 1 = Space complexity: O(1) the only extra variables are i, j and t n 1 (n) 2 = O n 2 16
Inversions Given an array a[n], the number of inversions is defined as number of pairs (i, j) where i < j but a[i] > a[j] Examples: (1, 2, 3, 4, 5) 0 inversions (1, 2, 3, 5, 4) 1 inversion: (3, 4) (1, 2, 5, 3, 4) 2 inversions: (2, 3), (2, 4) (2, 1, 4, 3, 6, 5) 3 inversions: (0, 1), (2, 3), (4, 5) (5, 4, 3, 2, 1) 10 inversions: (0, 1), (0, 2), (0, 3), (0, 4), (1, 2),..., (3, 4) (4, 3, 1, 2, 4, 1) how many inversions? 17
Inversions The number of inversions is also exactly the number of minimum number of adjacent pairwise swaps required to swap an array, i.e. the actual number of swaps performed by Bubble Sort Zero: Sorted Close to zero: almost sorted n 1 (n) 2 : reverse order 18
Properties of Bubble Sort Relevant Tasks: D802 Bubble Sort Able to detect whether the array is already sorted The array is already sorted if no swaps is performed in the first pass If we implement this, the best-case time complexity becomes O(n) In practice, Bubble Sort is slower than Selection Sort because of the number of writes to memory 19
Insertion Sort ( 插入排序法 ) Initially you are have no cards Cards would be given to you one by one All you need to do is to add the new card to the correct, sorted, position: If the new card is x, move all cards larger than x to the right by 1 position Insert x into the vacated position 1 3 5 7 4 1 3 -- (5) 5 7 1 3 4 5 7 20
Insertion Sort In each pass, use a separate variable t tostore a[i] Count j from i-1 down to 0 If a[j] > t, overwrite a[j + 1] Otherwise, break a[j + 1] is the correct position to insert t If t is the smallest element, j will become -1, and therefore t will be inserted into a[0] Round a[0] a[1] a[2] a[3] a[4] a[5] i = 0 2 3 6 5 1 4 i = 1, j = 0 2 t=3 6 5 1 4 i = 1, insert 2 3 6 5 1 4 i = 2, j = 1 2 3 t=6 5 1 4 i = 2, insert 2 3 6 5 1 4 i = 3, j = 2 2 3 6 t=5 1 4 i = 3, j = 1 2 3 --(6) 6 1 4 i = 3, insert 2 3 5 6 1 4 i = 4, j = 3 2 3 5 6 t=1 4 i = 4, j = 2 2 3 5 --(6) 6 4 i = 4, j = 1 2 3 --(5) 5 6 4 i = 4, j = 0 2 --(3) 3 5 6 4 i = 4, insert 1 2 3 5 6 4 i = 5, j = 4 1 2 3 5 6 t=4 i = 5, j = 3 1 2 3 5 --(6) 6 i = 5, j = 2 1 2 3 --(5) 5 6 i = 5, insert 1 2 3 4 5 6
Insertion Sort Warning: for j := i 1 downto 0 do is WRONG (Please refer to HKOI 2016/17 Heat FITB Q1) t is the new number Place t at a[j + 1] j=1 i=4 t=4 1 3 5 7 4 1 3 5 5 7 1 3 4 5 7 22
Insertion Sort Analysis Q: What is the actual number of comparisons performed? 23
Properties of Insertion Sort When the array is almost sorted, Insertion Sort will run quite fast The required number of "movements" a[j + 1] = a[j] is exactly the number of inversions For example, (2, 1, 4, 3, 6, 5) only requires 3 movements Therefore, the best-case time complexity is O(n), while the average-case and worst-case time complexity is O(n 2 ) Space complexity: O(1) 24
Shell Sort As discussed earlier, each "move" in Bubble Sort and Shell Sort reduces the number of inversions by 1 Shell Sort is a modification to Bubble Sort or Insertion Sort: Reduce a greater number (>=1) of inversions in each step In Bubble Sort and Insertion Sort, we considered adjacent elements In Shell Sort, we have multiple, decreasing "gap" values For example, n = 100, then gaps could be (64, 32, 16, 8, 4, 2, 1) Any gap sequence is ok as long as the last gap value is 1 The original Bubble / Insertion Sort has a single gap value: (1) 25
Shell Sort Gap = 3 4 1 5 2 6 3 2 1 5 4 6 3 2 1 5 4 6 3 Inversions 6 5 5 Gap = 1 2 1 3 4 6 5 1 2 3... 4 6 5 1 2 3 4 6 5 1 2 3 4 5 6 2 1 1 0 Prove that the number of inversion won't increase with such swappings 26
Shell Sort Implementation Gap = n, n,, 1 2 4 e.g. n = 100, gap = (50, 25, 12, 6, 3, 1) Performance depends on n, the array and gap 27
Counting Sort ( 計數排序法 ) You are playing scrabble and you have n = 10000 letter tiles Sort the tiles from A to Z We can create 26 piles: pile A, pile B,, to pile Z Process tiles one by one by placing it to the suitable pile Stack the piles together from pile A to Z C E M A T E A A O A A A C E E M O T frequency array A 3 B 0 C 1 D 0 E 2 28
Counting Sort The size of the frequency array has to be at least maxvalue + 1 Print i freq[i] times Example storing instead of printing directly: (j is a counter) 29
Counting Sort - Analysis To construct the frequency array, each element has to be considered exactly once n operations Loop over the range ( 範圍 ) of possible values r (range) operations Within the loop, the inner for loop will run for n times The best, average and worst-case time complexity is O n + r Space complexity is O(r) Therefore, Counting Sort is most suitable when r is large and r is small Example: n = 1000000, r = 100000 30
Radix Sort Sort n integers. Each integer has w digits For digit i = 0 (least-significant) to w 1 (most significant) Prepare 10 lists, one for each digit 0, 1, 2,..., 9 Loop through the array: If the i-th digit of a number is x, insert it into list x Concatenate the 10 lists to form the new array for the next step 31
Radix Sort example Integers to sort: 477 251 671 532 237 401 602 335 (n = 8, w = 3) Step i = 0 (units digit) 0 1 2 3 4 5 6 7 8 251 532 335 477 671 602 237 401 9 Result: 251 671 401 532 602 335 477 237 32
Radix Sort example From previous step: 251 671 401 532 602 335 477 237 Step i = 1 (tens digit) 0 1 2 3 4 5 6 7 8 401 532 251 671 602 335 477 237 9 Result: 401 602 532 335 237 251 671 477 33
Radix Sort example From previous step: 401 602 532 335 237 251 671 477 Step i = 2 (hundreds digit) 0 1 2 3 4 5 6 7 8 237 335 401 532 602 251 477 671 9 Result: 237 251 335 401 477 532 602 671 34
Radix Sort - Analysis Time complexity: O(nw) Space complexity: O(n + m) where m is the number of "buckets" 35
Number of ways to arrange numbers If you are given n different numbers, how many ways can you arrange them in different orders? For example, n = 4, there are 24 ways (permutations 排列 ) 1 2 3 4 2 1 3 4 3 1 2 4 4 1 2 3 1 2 4 3 2 1 4 3 3 1 4 2 4 1 3 2 1 3 2 4 2 3 1 4 3 2 1 4 4 2 1 3 1 3 4 2 2 3 4 1 3 2 4 1 4 2 3 1 1 4 2 3 2 4 1 3 3 4 1 2 4 3 1 2 1 4 3 2 2 4 3 1 3 4 2 1 4 3 2 1 Answer: n n 1 n 2 2 1 = n! ways 36
Lower bound for comparison based sorting If there is one thing that has x possibilities and you have to find out which one is it, how many "Yes-No questions" do you need to ask? Ideally, each question should eliminate half of the remaining possibilities Number of questions required = lg x Note: lg is log 2 37
Lower bound for comparison based sorting lg n! = lg n n 1 2 1 = lg (1 n) 2 (n 1 ) lg n n = lg n n 2 = (n lg n)/2 n/2 terms lg n! = lg n n 1 2 1 lg n n = lg n n = n lg n n terms Therefore, (n lg n)/2 lg n! n lg n lg n! = O(n lg n) 38
Quick Sort and Merge Sort Divide and conquer approach Divide Conquer Divide the array into 2 parts, and recursively sort each part Combine the two sorted arrays into one large sorted array Base case An array of length 0 or 1 must be sorted 39
Quick Sort Using a pivot ( 支點 ) p, separate the array into two smaller arrays One containing all values < p, another > p Original array: Let's choose the first element (6) as pivot 6 2 1 9 3 5 4 8 7 Divide: split the array into 2, and sort them 2 1 3 5 4 6 9 8 7 smaller than 6 larger than 6 1 2 3 4 5 6 7 8 9 Haskell implementation of Quick Sort Conquer: Combine into one 40
Quick Sort Task: D807 Lomuto partition scheme: maintain length of the part <= p <=p >p x unsorted p swap Case 1: x <= p Case 2: x > p Finally: <=p x >p unsorted p <=p >p unsorted p <=p >p p swap <=p p >p Place the pivot at the correct position 41
Quick Sort Worst Case Task: D808 Sorted Reverse Sorted Equal Elements Pretty Random 8 1 2 3 4 5 6 7 8 > l = 1, r = 8 (8 elements) >> l = 1, r = 7 (7 elements) >>> l = 1, r = 6 (6 elements) >>>> l = 1, r = 5 (5 elements) >>>>> l = 1, r = 4 (4 elements) >>>>>> l = 1, r = 3 (3 elements) >>>>>>> l = 1, r = 2 (2 elements) >>>>>>>> l = 1, r = 1 (1 element) 1 2 3 4 5 6 7 8 36 8 8 7 6 5 4 3 2 1 > l = 1, r = 8 (8 elements) >> l = 2, r = 8 (7 elements) >>> l = 2, r = 7 (6 elements) >>>> l = 3, r = 7 (5 elements) >>>>> l = 3, r = 6 (4 elements) >>>>>> l = 4, r = 6 (3 elements) >>>>>>> l = 4, r = 5 (2 elements) >>>>>>>> l = 5, r = 5 (1 element) 1 2 3 4 5 6 7 8 36 8 0 0 0 0 0 0 0 0 > l = 1, r = 8 (8 elements) >> l = 1, r = 7 (7 elements) >>> l = 1, r = 6 (6 elements) >>>> l = 1, r = 5 (5 elements) >>>>> l = 1, r = 4 (4 elements) >>>>>> l = 1, r = 3 (3 elements) >>>>>>> l = 1, r = 2 (2 elements) >>>>>>>> l = 1, r = 1 (1 element) 0 0 0 0 0 0 0 0 36 8 6 7 5 2 3 8 1 4 > l = 1, r = 8 (8 elements) >> l = 1, r = 3 (3 elements) >>> l = 2, r = 3 (2 elements) >>>> l = 3, r = 3 (1 element) >> l = 5, r = 8 (4 elements) >>> l = 5, r = 5 (1 elements) >>> l = 7, r = 8 (2 elements) >>>> l = 7, r = 7 (1 element) 1 2 3 4 5 6 7 8 22 42
Quick Sort Hoare partition scheme: maintain lengths of both parts p <=p unsorted >=p Step 1: expand p <=p <p >=p unsorted <=p >p p <=p >=p unsorted <=p >=p >=p Step 2: swap p <=p <=p unsorted >=p >=p Repeat until no unsorted part remains p <=p Recursively sort both parts (including p) >=p Advantage: Handle equal elements more efficiently 43
Quick Sort Time complexity: Best case: O(n lg n) Average case: O(n lg n) Worst case: O(n 2 ) Ways to lowering the chance of uneven splits Most common: Choose a pivot randomly (pseudo-randomly) e.g. pivot = a[(lo + hi) / 2] or a[lo + rand() % (hi - lo + 1)] Random shuffle the array first Choose the pivot using median of medians Space complexity: O(lg n) worst case: O(n) 44
Merge Sort Split an array a[lo.. hi] into two halves and recursively sort them a[lo.. mid], and a[mid + 1.. hi] Completely avoids uneven splits 45
Merge Sort unsorted Split: Recursively sort: Merge: Case 1: x <= y x temporary sorted unsorted sorted temporary y sorted unsorted sorted Compare the smallest (left-most) element of each part Case 2: x > y temporary x sorted y sorted x sorted y sorted Store: temporary sorted 46
Merge Sort Time complexity: Best, average and worst case: O(n lg n) Space complexity: O n Worse than most other algorithms 47
Sorting using C++ standard template library sort(begin, end); Sorts the range [begin, end) so begin should be the pointer to the first element and end be the pointer to one after the last element 48
M0713 Cream Soda Find the median of n numbers Here, it is guaranteed that n is odd a[0] a[1] a[2] a[3] a[4] Before Sorting 7 5 6 9 8 After Sorting 5 6 7 8 9 Where is the answer (median)? 49
Quick Select We actually only care about the median It is a waste to sort the other elements We modify our Quick Sort algorithm Only recursively sort the part that contains the position we care about position we care about <=p p >p Sort this part No need to sort this part <=q q >q No need to sort this part Sort this part 50
Quick Select Time complexity: O(n) Space complexity: O(lg n) 51
Stability of Sorting A sorting algorithm is stable if the original order is preserved among elements with the same sorting key Here sorting key is price. A stable sort must retain Banana before Candy and Egg before Pork before Fish Stable sort must return Unstable sort may return Item Price Item Price Item Price Apple 5.5 Apple 5.5 Apple 5.5 Banana 12 Banana 12 Candy 12 Egg 25 sort Candy 12 Banana 12 Rice 70 Egg 25 Egg 25 Pork 25 Pork 25 Fish 25 Fish 25 Fish 25 Pork 25 Candy 12 Rice 70 Rice 70 52
Stability of Sorting stable_sort(a, a + n); Stable sort is not useful when the values are distinct Useful when sorting multi key data 53
Stability of Sorting Which sorting algorithms discussed so far are stable? Stable Insertion Sort Bubble Sort Counting Sort (LSD) Radix Sort Merge Sort Unstable Selection Sort Shell Sort Quick Sort 54
01068 Moliu Sorting So how to perform multi-key sorting? Method 1: Custom comparison function ( 自訂排序 ) with sorter array a Set sorter array a[i] = i sort(a, a + n, cmp) cmp is the name of the comparison function, which should return a bool: true when the first argument should be placed before the second argument, false otherwise bool cmp(const T& a, const T& b) 55
01068 Moliu Sorting Method 2: pair with custom comparison function Data in a pair will be moved together 56
01068 Moliu Sorting Method 3: Use default comparison for pair: first is compared first, followed by second 57
Binary Search Once we have a sorted array, we can use the binary search algorithm to find If an element exists in an array Its position if it exists It is does not exists, the position it should be inserted into The smallest element >= x / > x The largest element <= x / < x 58
Binary Search Find the position of 5 in the array (or find the left-most element >= 5) 1 3 3 4 6 7 7 8 9 9 10 11 12 15 18 lo mid hi 1 3 3 4 6 7 7 8 9 9 10 11 12 15 18 lo mid hi 1 3 3 4 6 7 7 8 9 9 10 11 12 15 18 mid hi lo 1 3 3 4 6 7 7 8 9 9 10 11 12 15 18 lmh 1 3 3 4 6 7 7 8 9 hi 9 is the 10 position 11 12 of the 15 18 hi lo right-most Sorting and element Searching < 5 59
Binary Search Memorize these correct implementations ; 60
Binary Search - Analysis Search for x in the the range [begin, end) so begin should be the pointer to the first element and end be the pointer to one after the last element Time complexity: O(lg n) Space complexity: O(1) binary_search(begin, end, x) returns true / false whether x is present lower_bound(begin, end, x) returns the pointer to left-most element >= x upper_bound(begin, end, x) returns the pointer to left-most element > x equal_range(begin, end, x) returns a pair of pointers: left-most element >= x, left-most element > x When x is larger than every element, the pointer end is returned 61
Binary Search on Answer If a tasks asks you to find max possible value such that... Assume that the answer is x, then you can binary search for x if: There is an efficient way to check whether a value v is possible It is possible for all x x; and it is impossible for all x > x possible x impossible Vice versa, you can binary search for minimum possible value if impossible x possible 62
M1023 Seating Plan Given an array a[1..n], choose m elements such that the minimum absolute difference is maximized Example: m = 3 Assume that the answer is 6 1 9 15 17 21 25 30 1 9 15 17 21 25 30 Able to choose 5 >= 3 elements Possible: the answer could be larger >= 6 >= 6 >= 6 >= 6 Assume that the answer is 16 1 9 15 17 21 25 30 >= 16 >= 16 >= 6 Unable to choose 3 elements Impossible: the answer must be smaller 63
M1023 Seating Plan Determine min and max answer Write check(x) function to determine whether x is possible Adjust lo and hi possible hi lo x impossible Time complexity: O(f(n) lg (hi lo)) 64
Ternary Search y = f(x) If you are given a function y = f(x) and you are asked to find a such that y is minimized, then you can use ternary search if: f x 1 f x < f(x) for all lb x < a (to the left of a) < f x + 1 for all a < x ub (to the right of a) Note: lb and ub are the lower bound and upper bound of the answer Note the strictly < condition Exception: f x = f x + 1 = minimum is acceptable minimum point: a 65
Ternary Search Note that we are discussing discrete functions Functions that can perform ternary search on: 8 7 6 5 3 2 5 9 10 12 9 6 5 4 2 2 2 3 4 5 Function that cannot perform ternary search on: 8 7 6 6 6 4 5 6 7 8 66
Ternary Search Pick two points between lo and hi, there are two variants: Three parts p = (lo * 2 + hi) / 3 q = (lo + hi * 2) / 3 Two parts p = (lo + hi) / 2 q = (lo + hi) / 2 + 1 lo p q hi lo p q hi f(p) > f(q): answer is between p and hi f(p) >= f(q): answer is between q and hi f(p) == f(q): answer is between p and q f(p) < f(q): answer is between lo and p f(p) < f(q): answer is between lo and q 67
Master Theorem Time complexity analysis of divide and conquer algorithms The time required for a problem size of n = T(n) T(n) = a T( n ) + f(n) b 3 cases: (simplified) f n < O(n log b a ) then T n = O(n log b a ) f n = O(n log b a log k n) then T n = O n log b a log k+1 n f n > O(n log b a ) then T n = O f(n) 68
Master Theorem Examples Binary Search: T(n) = T( n 2 ) + O(1) Case 2: O 1 = O n log 2 1 log 0 n Therefore T n = O n log 2 1 log 0+1 n = O(lg n) Merge Sort: T(n) = 2T( n 2 ) + O(n) Case 2: O n = O n log 2 2 log 0 n Therefore T n = O n log 2 2 log 0+1 n = O(n lg n) Quick Select: T(n) = T( n 2 ) + O(n) Case 3: O n > O n log 2 1 Therefore T n = O f n = O n L-Pieces: T(n) = 4T( n 2 ) + O 1 (assume n = side length) Case 1: f n < O(n log b a ) then T n = O(n log b a ) Case 2: f n = O(n log b a log k n) then T n = O n log b a log k+1 n Case 3: f n > O(n log b a ) then T n = O f(n) Case 1: O 1 < O n log 2 4 Therefore T n = O(n log 2 4 ) = O n 2 69