ITEC2620 Introduction to Data Structures Lecture 6a Complexity Analysis Recursive Algorithms Complexity Analysis Determine how the processing time of an algorithm grows with input size What if the algorithm is recursive? Processing time is based on time to process sub-problems Use recurrence relations Recurrence Relations The problem is solved through smaller sub-problems The time to solve a problem is based on the time to solve smaller subproblems Express the time to solve a problem of size n as T(n) From Recursion to Recurrence Relations Recursion has two cases Recursive case Base case Recurrence relations need two cases Recurring case Base case Usually T(1) = 1 Eliminate constant factors 1
Factorial Example I Recursive formulation n! = n * n-1! if n > 1 Recursive sub-problem n! = 1 if n = 1, 0 Base problem Factorial Example II Building the recurrence relations n! = n * n-1! if n > 1 The time to do a problem of size n is the time to do a problem of size n-1 + a multiplication n! = 1 if n = 1, 0 The time to do a problem of size 1 is 1 (a constant) Factorial Example III Recurrence relations The time to do a problem of size n T(n) The time to do a problem of size n-1 T(n-1) A multiplication 1 T(n) = T(n-1) + 1 T(1) = 1 Binary Search Example (Worst Case) I Do binary search on the remaining half to search Recursive sub-problem Return failure (-1) Base problem 2
Binary Search Example (Worst Case) II Building the recurrence relations Do binary search on the remaining half The time to do a problem of size n is the time to do a problem of size n/2 + a comparison Return failure (-1) The time to do a problem of size 1 is 1 (a constant) Binary Search Example (Worst Case) III Recurrence relations The time to do a problem of size n T(n) The time to do a problem of size n/2 T(n/2) A comparison 1 T(n) = T(n/2) + 1 T(1) = 1 Solving Recurrence Relations By expansion Start with T(n) and expand out terms mathematically Reduce into formula form Numerically Start with T(1) and calculate values Make a formula for the number series Factorial Example I By expansion T(n) = T(n-1) + 1 = {T(n-2) + 1} + 1 = {[T(n-3) + 1] + 1} + 1 = {[T(1) + 1] + 1 + + 1} + 1 T(n) = n * 1 = O(n) 3
Factorial Example II Numerically T(1) = 1 T(2) = T(1) + 1 = 2 T(3) = T(2) + 1 = 3 T(4) = T(3) + 1 = 4 T(n) = T(n-1) + 1 = n = O(n) Factorial Example III Numerically is usually much easier Both give O(n) as expected Note: non-recursive is also O(n) How much time does it take to multiply n numbers? Binary Search Example Numerically T(1) = 1 T(2) = T(1) + 1 = 2 T(4) = T(2) + 1 = 3 T(8) = T(4) + 1 = 4 T(n) = T(n/2) + 1 = logn + 1 = O(logn) Quicksort I What is the complexity of Quicksort? Is there a Best, Worst, and Average case? Can Quicksort run faster or slower based on the input? Yes! 4
Quicksort II What is the best case? Pivot always makes two equal divisions What is the worst case? Pivot never makes new divisions All of the elements are always to one side What is the average case? Pivot makes roughly equal divisions Quicksort III Best Case Build the recurrence relations The time to do a problem of size is the time to do sub-problems of size + (actions) The time to do a problem of size 1 is 1 (a constant) Quicksort IV We are sorting n elements We want to know the time to sort n elements The time to do a problem of size n Quicksort V Pivot always makes two equal divisions 2 sub-problems Of size n/2 Creating divisions requires comparing all of the other elements to see if they are larger or smaller + n compares (during partition) 5
Quicksort VI The time to do a problem of size n is the time to do 2 sub-problems of size n/2 + n compares (during partition) T(n) = 2 * T(n/2) + n The time to do a problem of size 1 is 1 (a constant) T(1) = 1 Quicksort VII Numerically T(1) = 1 = 1 * 1 T(2) = 2 * T(1) + 2 = 4 = 2 * 2 T(4) = 2 * T(2) + 4 = 12 = 3 * 4 T(8) = 2 * T(4) + 8 = 32 = 4 * 8 T(16) = 2 * T(8) + 16 = 80 = 5 * 16 Quicksort VIII T(1) = 1 = 1* 1 T(2) = 2 * T(1) + 2 = 4 = 2 * 2 T(4) = 2 * T(2) + 4 = 12 = 3 * 4 T(8) = 2 * T(4) + 8 = 32 = 4 * 8 T(16) = 2 * T(8) + 16 = 80 = 5 * 16 T(n) = 2 * T(n/2) + n = (logn + 1) * n = O(nlogn) Quicksort IX Worst Case Build the recurrence relations The time to do a problem of size is the time to do sub-problems of size + (actions) The time to do a problem of size 1 is 1 (a constant) 6
Quicksort X We are sorting n elements We want to know the time to sort n elements The time to do a problem of size n Quicksort XI Pivot never makes new divisions 1 sub-problem Of size n-1 Not creating new divisions still requires comparing all of the other elements to see if they are larger or smaller + n compares (during partition) Quicksort XII The time to do a problem of size n is the time to do 1 sub-problem of size n-1 + n compares (during partition) T(n) = T(n-1) + n The time to do a problem of size 1 is 1 (a constant) T(1) = 1 Quicksort XIII By expansion T(n) = T(n-1) + n = {T(n-2) + n-1} + n = {[T(n-3) + n-2] + n-1} + n T(n) = n i i 1 = O(n 2 ) n( n 1) 2 7
Quicksort XIV Average case O(nlogn) 14.2.4 if you don t believe me Mergesort I What is the complexity of Mergesort? Is there a Best, Worst, and Average case? Can Mergesort run faster or slower based on the input? No! Mergesort II Does Mergesort look at the values before splitting? No Does merge always process all values? Yes Complexity of Mergesort does not depend on input Mergesort III Build the recurrence relations The time to do a problem of size is the time to do sub-problems of size + (actions) The time to do a problem of size 1 is 1 (a constant) 8
Mergesort IV We are sorting n elements We want to know the time to sort n elements The time to do a problem of size n Mergesort V Always split into two equal divisions 2 sub-problems Of size n/2 Merging divisions requires comparing an element from each division + n compares (during merge) Mergesort VI The time to do a problem of size n is the time to do 2 sub-problems of size n/2 + n compares (during merge) T(n) = 2 * T(n/2) + n The time to do a problem of size 1 is 1 (a constant) T(1) = 1 Mergesort VII Numerically T(1) = 1 = 1* 1 T(2) = 2 * T(1) + 2 = 4 = 2 * 2 T(4) = 2 * T(2) + 4 = 12 = 3 * 4 T(8) = 2 * T(4) + 8 = 32 = 4 * 8 T(16) = 2 * T(8) + 16 = 80 = 5 * 16 T(n) = 2 * T(n/2) + n = (logn + 1) * n = O(nlogn) 9
Summary I Mergesort and Quicksort both have O(nlogn) in the best and average cases Mergesort is also O(nlogn) in the worst case Quicksort is O(n 2 ) in the worst case Summary II Worst case is unlikely, and constants are smaller for Quicksort n/2-1 in quicksort n/2 in mergesort Quicksort is the preferred algorithm for many applications But not all! Readings and Assignments Suggested Readings from Shaffer (third edition) 2.4, 14.2.2 10