CS 4407 Algorithms Lecture 3: Iterative and Divide and Conquer Algorithms Prof. Gregory Provan Department of Computer Science University College Cork 1
Lecture Outline CS 4407, Algorithms Growth Functions Mathematical specification of growth functions Iterative Algorithms Divide-and-Conquer Algorithms
Today s Learning Objectives Describe mathematical principles for specifying the growth of run-time of an algorithm Classify the growth functions Q, O, W, o, w Iterative Algorithm Analysis Divide-and-Conquer Algorithm Analysis CS 4407, Algorithms
CS 4407, Algorithms Analyzing Algorithms Assumptions generic one-processor, random access machine running time (others: memory, communication, etc) Worst Case Running Time: the longest time for any input of size n upper bound on the running time for any input in some cases like searching, this is close Average Case Behavior: the expected performance averaged over all possible inputs it is generally better than worst case behavior sometimes it s roughly as bad as worst case
CS 4407, Algorithms Notation: Growth Functions Theta f(n) = θ(g(n)) f(n) c g(n) BigOh f(n) = O(g(n)) f(n) c g(n) Omega f(n) = Ω(g(n)) f(n) c g(n) Little Oh f(n) = o(g(n)) f(n) << c g(n) Little Omega f(n) = ω(g(n)) f(n) >> c g(n)
Q-notation For a given function g(n), we denote by Q(g(n)) the set of functions Q(g(n)) = {f(n): there exist positive constants c 1, c 2 and n 0 such that 0 c 1 g(n) f(n) c 2 g(n), for all n n 0 } We say g(n) is an asymptotically tight bound for f(n) CS 4407, Algorithms
O-notation For a given function g(n), we denote by O(g(n)) the set of functions O(g(n)) = {f(n): there exist positive constants c and n 0 such that 0 f(n) cg(n), for all n n 0 } We say g(n) is an asymptotic upper bound for f(n) CS 4407, Algorithms
W-notation For a given function g(n), we denote by W(g(n)) the set of functions W(g(n)) = {f(n): there exist positive constants c and n 0 such that 0 cg(n) f(n) for all n n 0 } We say g(n) is an asymptotic lower bound for f(n) CS 4407, Algorithms
CS 4407, Algorithms Relations Between Q, W, O For any two functions g(n) and f(n), f(n) = Q(g(n)) if and only if f(n) = O(g(n)) and f(n) = W(g(n)). i.e., Q(g(n)) = O(g(n)) W(g(n)).
Complexity of Simple Algorithms Simple Interation Dealing with Loops Loop invariant method Divide and Conquer Algorithms CS 4407, Algorithms
CS 4407, Algorithms Iterative Algorithm Running Time T(n), or the running time of a particular algorithm on input of size n, is taken to be the number of times the instructions in the algorithm are executed. Pseudo code algorithm illustrates the calculation of the mean (average) of a set of n numbers: 1. n = read input from user 2. sum = 0 3. i = 0 4. while i < n 5. number = read input from user 6. sum = sum + number 7. i = i + 1 8. mean = sum / n The computing time for this algorithm in terms on input size n is: T(n) = 4n + 5. Statement Number of times executed 1 1 2 1 3 1 4 n+1 5 n 6 n 7 n 8 1
Analysis of Simple Programs (no recursion) Sum the costs of the lines of the program Compute the bounding function e.g., O(*) CS 4407, Algorithms
Analysing Loops Use Loop Invariants Intuitive notion and example CS 4407, Algorithms
CS 4407, Algorithms Designing an Algorithm Define Problem Define Loop Invariants Define Measure of Progress 79 km to school Define Step Define Exit Condition Maintain Loop Inv Exit Exit Make Progress Initial Conditions Ending Exit 79 km 75 km km 0 km Exit Exit
CS 4407, Algorithms Typical Loop Invariant If the input consists of an array of objects I have a solution for the first i objects. i objects Exit 79 km 75 km i to i+1 Extend the solution into a solution for the first i+1. Exit Done when solution for n
Typical Loop Invariant CS 4407, Algorithms If the output consists of an array of objects I have an output produced for the first i objects. i objects Produce the i+1-st output object. Exit 79 km 75 km i to i+1 Exit Done when output n objects.
Binary search Example of Approach Standard algorithm Input Sorted list, and key Goal: find key in list CS 4407, Algorithms
Define Problem: Binary Search PreConditions Key 25 Sorted List 3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95 PostConditions Find key in list (if there). 3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95 CS 4407, Algorithms
Define Loop Invariant Maintain a sublist Such that key 25 3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95 CS 4407, Algorithms
Define Loop Invariant Maintain a sublist. If the key is contained in the original list, then the key is contained in the sublist. key 25 3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95 CS 4407, Algorithms
Make Progress Define Step Maintain Loop Invariant key 25 3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95 CS 4407, Algorithms
Define Step Cut sublist in half. Determine which half key would be in. Keep that half. key 25 3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95 CS 4407, Algorithms
Define Step Cut sublist in half. Determine which half the key would be in. Keep that half. key 25 3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95 If key mid, then key is in left half. If key > mid, then key is in right half. CS 4407, Algorithms
Define Step It is faster not to check if the middle element is the key. Simply continue. key 43 3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95 If key mid, then key is in left half. If key > mid, then key is in right half. CS 4407, Algorithms
CS 4407, Algorithms Make Progress Exit The size of the list becomes smaller. 79 km 75 km 3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95 79 km 3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95 75 km
Initial Conditions km key 25 3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95 n km The sublist is the entire original list. If the key is contained in the original list, then the key is contained in the sublist. CS 4407, Algorithms
CS 4407, Algorithms Ending Algorithm Exit key 25 3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95 0 km If the key is contained in the original list, then the key is contained in the sublist. Sublist contains one element. Exit If the key is contained in the original list, then the key is at this location.
If key not in original list If the key is contained in the original list, then the key is contained in the sublist. Loop invariant true, even if the key is not in the list. key 24 3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95 If the key is contained in the original list, then the key is at this location. Conclusion still solves the problem. Simply check this one location for the key. CS 4407, Algorithms
Running Time The sublist is of size n, n / 2, n / 4, n / 8,,1 Each step (1) time. Total = (log n) key 25 3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95 If key mid, then key is in left half. CS 4407, Algorithms If key > mid, then key is in right half.
Code CS 4407, Algorithms
Divide and Conquer Algorithms Well-known class of algorithm Requires special approach for complexity analysis CS 4407, Algorithms
Divide-and-Conquer Analysis The analysis of divide and conquer algorithms require us to solve a recurrence. Recurrences are a major tool for analysis of algorithms CS 4407, Algorithms
CS 4407, Algorithms MergeSort A L G O R I T H M S A L G O R I T H M S divide cn T(n/2) T(n/2)
CS 4407, Algorithms MergeSort A L G O R I T H M S A L G O R I T H M S Divide #1 A L G O R I T H M S Divide #2 cn T(n/2) T(n/2) T(n/4) T(n/4) T(n/4) T(n/4)
MergeSort Solve T(n) = T(n/2) + T(n/2) + cn cn (n/2) +c (n/2) +c T(n/4) T(n/4) T(n/4) T(n/4) Recurrence T ( n) 2T c n cn 2 CS 4407, Algorithms n n 1 1
CS 4407, Algorithms Recursion-tree method A recursion tree models the costs (time) of a recursive execution of an algorithm. The recursion tree method is good for generating guesses for the substitution method. The recursion-tree method can be unreliable, just like any method that uses ellipses ( ). The recursion-tree method promotes intuition, however.
CS 4407, Algorithms Example of recursion tree Solve T(n) = T(n/4) + T(n/2) + n 2 :
CS 4407, Algorithms Example of recursion tree Solve T(n) = T(n/4) + T(n/2) + n 2 : T(n)
CS 4407, Algorithms Example of recursion tree Solve T(n) = T(n/4) + T(n/2) + n 2 : n 2 T(n/4) T(n/2)
CS 4407, Algorithms Example of recursion tree Solve T(n) = T(n/4) + T(n/2) + n 2 : n 2 (n/4) 2 (n/2) 2 T(n/16) T(n/8) T(n/8) T(n/4)
CS 4407, Algorithms Example of recursion tree Solve T(n) = T(n/4) + T(n/2) + n 2 : n 2 (n/4) 2 (n/2) 2 (n/16) 2 (n/8) 2 (n/8) 2 (n/4) 2 Q(1)
CS 4407, Algorithms Example of recursion tree Solve T(n) = T(n/4) + T(n/2) + n 2 : n 2 n 2 (n/4) 2 (n/2) 2 (n/16) 2 (n/8) 2 (n/8) 2 (n/4) 2 Q(1)
CS 4407, Algorithms Example of recursion tree Solve T(n) = T(n/4) + T(n/2) + n 2 : n 2 (n/4) 2 (n/2) 2 (n/16) 2 (n/8) 2 (n/8) 2 (n/4) 2 n2 5 n 2 16 Q(1)
CS 4407, Algorithms Example of recursion tree Solve T(n) = T(n/4) + T(n/2) + n 2 : n 2 (n/4) 2 (n/2) 2 (n/16) 2 (n/8) 2 (n/8) 2 (n/4) 2 n2 5 n 2 16 25 n 2 256 Q(1)
CS 4407, Algorithms Example of recursion tree Solve T(n) = T(n/4) + T(n/2) + n 2 : n 2 (n/4) 2 (n/2) 2 (n/16) 2 (n/8) 2 (n/8) 2 (n/4) 2 n2 5 n 2 16 25 n 2 256 Q(1) Total = n 2 1 16 5 16 5 2 16 5 3 = Q(n 2 ) geometric series
CS 4407, Algorithms Solution: geometric series 1 1 1 2 x x x for x < 1 1 1 1 1 2 x x x x x n n for x 1
CS 4407, Algorithms The master method The master method applies to recurrences of the form T(n) = a T(n/b) + f (n), where a 1, b > 1, and f is asymptotically positive.
CS 4407, Algorithms Idea of master theorem Recursion tree: f (n) a f (n/b) f (n/b) f (n/b) a h = log b n f (n/b 2 ) f (n/b 2 ) f (n/b 2 ) f (n) a f (n/b) a 2 f (n/b 2 ) T (1) #leaves = a h = a log bn = n log ba n log ba T (1)
Three common cases Compare f (n) with n log ba : 1. f (n) = O(n log ba e ) for some constant e > 0. f (n) grows polynomially slower than n log ba (by an n e factor). Solution: T(n) = Q(n log ba ). # leaves in recursion tree CS 4407, Algorithms
CS 4407, Algorithms Idea of master theorem Recursion tree: f (n) a f (n/b) f (n/b) a h = log b n f (n/b 2 ) f (n/b 2 ) f (n/b 2 ) f (n/b) f (n) a f (n/b) a 2 f (n/b 2 ) T (1) CASE 1: The weight increases n log ba geometrically from the root to the T (1) leaves. The leaves hold a constant fraction of the total weight. Q(n log ba )
Three common cases Compare f (n) with n log ba : 2. f (n) = Q(n log ba lg k n) for some constant k 0. f (n) and n log ba grow at similar rates. Solution: T(n) = Q(n log ba lg k+1 n). CS 4407, Algorithms
CS 4407, Algorithms Idea of master theorem Recursion tree: f (n) a f (n/b) f (n/b) a h = log b n f (n/b 2 ) f (n/b 2 ) f (n/b 2 ) f (n/b) f (n) a f (n/b) a 2 f (n/b 2 ) T (1) CASE 2: (k = 0) The weight is approximately the same on each of the log b n levels. n log ba T (1) Q(n log ba lg n)
Three common cases (cont.) Compare f (n) with n log ba : 3. f (n) = W(n log ba + e ) for some constant e > 0. f (n) grows polynomially faster than n log ba (by an n e factor), and f (n) satisfies the regularity condition that a f (n/b) c f (n) for some constant c < 1. Solution: T(n) = Q( f (n) ). CS 4407, Algorithms
CS 4407, Algorithms Idea of master theorem Recursion tree: f (n) a f (n/b) f (n/b) a h = log b n f (n/b 2 ) f (n/b 2 ) f (n/b 2 ) f (n/b) f (n) a f (n/b) a 2 f (n/b 2 ) T (1) CASE 3: The weight decreases geometrically from the root to the leaves. The root holds a constant fraction of the total weight. n log ba T (1) Q( f (n))
CS 4407, Algorithms Examples Ex. T(n) = 4T(n/2) + n a = 4, b = 2 n log ba = n 2 ; f (n) = n. CASE 1: f (n) = O(n 2 e ) for e = 1. T(n) = Q(n 2 ). Ex. T(n) = 4T(n/2) + n 2 a = 4, b = 2 n log ba = n 2 ; f (n) = n 2. CASE 2: f (n) = Q(n 2 lg 0 n), that is, k = 0. T(n) = Q(n 2 lg n).
CS 4407, Algorithms Examples Ex. T(n) = 4T(n/2) + n 3 a = 4, b = 2 n log ba = n 2 ; f (n) = n 3. CASE 3: f (n) = W(n 2 + e ) for e = 1 and 4(cn/2) 3 cn 3 (reg. cond.) for c = 1/2. T(n) = Q(n 3 ). Ex. T(n) = 4T(n/2) + n 2 /lg n a = 4, b = 2 n log ba = n 2 ; f (n) = n 2 /lg n. Master method does not apply. In particular, for every constant e > 0, we have n e w(lg n).