COMP 3170 - Analysis of Algorithms & Data Structures Shahin Kamali Lecture 4 - Jan. 14, 2019 CLRS 1.1, 1.2, 2.2, 3.1, 4.3, 4.5 University of Manitoba Picture is from the cover of the textbook CLRS. COMP 3170 - Analysis of Algorithms & Data Structures 1 / 16
Asymptotic Notations in a Nutshell Definition f(n) O(g(n)) M > 0, n 0 > 0 s.t. n > n 0, f (n) M g(n) Definition f(n) o(g(n)) M > 0, n 0 > 0 s.t. n > n 0, f (n) < M g(n) Definition f(n) Ω(g(n)) M > 0, n 0 > 0 s.t. n > n 0, f (n) M g(n) Definition f(n) ω(g(n)) M > 0, n 0 > 0 s.t. n > n 0, f (n) > M g(n) Definition f(n) Θ(g(n)) M 1, M 2 > 0, n 0 > 0 s.t. n > n 0, M 1 g(n) f (n) M 2 g(n) COMP 3170 - Analysis of Algorithms & Data Structures 2 / 16
Common Growth Rates Θ(1) constant complexity e.g., an algorithms that only samples a constant number of inputs Θ(log n) logarithmic complexity Binary search Θ(n) linear complexity Most practical algorithms :) Θ(n log n) pseudo-linear complexity Optimal comparison based sorting algorithms, e.g., merge-sort Θ(n 2 ) Quadratic complexity naive sorting algorithms (Bubble sort, insertion sort) Θ(n 3 ) Cubic Complexity naive matrix multiplication Θ(2 n ) Exponential Complexity The algorithm terminates but the universe is likely to end much earlier even if n 1000. COMP 3170 - Analysis of Algorithms & Data Structures 3 / 16
Techniques for Comparing Growth Rates Assume the running time of two algorithms are given by functions f (n) and g(n) and let f (n) L = lim n g(n) Then. o(g(n)) if L = 0 f (n) Θ(g(n)) if 0 < L < ω(g(n)) if L = If the limit is not defined, we need another method Note that we cannot compare two algorithms using big O and Ω notations E.g., algorithm A can have complexity O(n 2 ) and algorithm B has complexity O(n 3 ). We cannot state that A is faster than B (why?) COMP 3170 - Analysis of Algorithms & Data Structures 4 / 16
Fun with Asymptotic Notations Compare the grow-rate of log n and n r where r is a positive real number. COMP 3170 - Analysis of Algorithms & Data Structures 5 / 16
Fun with Asymptotic Notations Prove that n(sin(n) + 2) is Θ(n). Use the definition since the limit does not exist Define n 0, M 1, M 2 so that n > n 0 we have M 1n(sin(n) + 2) n qm 2n(sin(n) + 2). M 1 = 1/3, M 2 = 1, n 0 = 1 work! COMP 3170 - Analysis of Algorithms & Data Structures 6 / 16
Fun with Asymptotic Notations The same relationship that holds for relative values of numbers hold for asymptotic. E.g., if f (n) O(g(n)) [f (n) is asymptotically smaller than or equal to g(n)], then we have g(n) Ω(f (n)) [g(n) is asymptotically larger than or equal to f (n)]. In order to prove f (n) Θ(g(n)), we often show that f (n) O(n) and f (n) Ω(g(n)). Similarly, we have transitivity in asymptotic notations: if f (n) O(g(n)) and g(n) O(h(n)), we have f (n) O(h(n)). Max rule: f (n) + g(n) Θ(max{f (n), g(n)}). E.g., 2n 3 + 8n 2 + 16n log n Θ(max{2n 3, 8n 2, 16n log n}) = Θ(n 3 ). COMP 3170 - Analysis of Algorithms & Data Structures 7 / 16
Fun with Asymptotic Notations What is the time complexity of arithmetic sequences? n 1 (a + di)= na + dn(n 1) Θ(n 2 ) 2 i=0 What about geometric sequence? n 1 a 1 r n Θ(1) if 0 < r < 1 ar i 1 r = na Θ(n) if r = 1 i=0 a r n 1 Θ(r n ) if r > 1 r 1 What about Harmonic sequence? H n = n i=1 1 i ln(n) + γ Θ(log n) (γ is a constant 0.577) COMP 3170 - Analysis of Algorithms & Data Structures 8 / 16
Loop Analysis Identify elementary operations that require constant time The complexity of a loop is expressed as the sum of the complexities of each iteration of the loop. Analyse independent loops separately, and then add the results (use maximum rules and simplify when possible). If loops are nested, start with the innermost loop and proceed outwards. COMP 3170 - Analysis of Algorithms & Data Structures 9 / 16
Example of Loop Analysis Algo1 (n) 1. A 0 2. for i 1 to n do 3. for j i to n do 4. A A/(i j) 2 5. A A 100 6. return sum COMP 3170 - Analysis of Algorithms & Data Structures 10 / 16
Example of Loop Analysis Algo2 (A, n) 1. max 0 2. for i 1 to n do 3. for j i to n do 4. X 0 5. for k i to j do 6. X A[k] 7. if X > max then 8. max X 9. return max COMP 3170 - Analysis of Algorithms & Data Structures 11 / 16
Example of Loop Analysis Algo3 (n) 1. X 0 2. for i 1 to n 2 do 3. j i 4. while j 1 do 5. X X + i/j 6. j j/2 7. return X COMP 3170 - Analysis of Algorithms & Data Structures 12 / 16
MergeSort Sorting an array A of n numbers Step 1: We split A into two subarrays: A L consists of the first n 2 elements in A and A R consists of the last n 2 elements in A. Step 2: Recursively run MergeSort on A L and A R. Step 3: After A L and A R have been sorted, use a function Merge to merge them into a single sorted array. This can be done in time Θ(n). COMP 3170 - Analysis of Algorithms & Data Structures 13 / 16
MergeSort MergeSort(A, n) 1. if n = 1 then 2. S A 3. else 4. n L n 2 5. n R n 2 6. A L [A[1],..., A[n L ]] 7. A R [A[n L + 1],..., A[n]] 8. S L MergeSort(A L, n L ) 9. S R MergeSort(A R, n R ) 10. S Merge(S L, n L, S R, n R ) 11. return S COMP 3170 - Analysis of Algorithms & Data Structures 14 / 16
Analysis of MergeSort The following is the corresponding sloppy recurrence (it has floors and ceilings removed): { 2 T ( ) n T (n) = 2 + cn if n > 1 d if n = 1. The exact and sloppy recurrences are identical when n is a power of 2. The recurrence can easily be solved by various methods when n = 2 j. The solution has growth rate T (n) Θ(n log n). It is possible to show that T (n) Θ(n log n) for all n by analyzing the exact recurrence. COMP 3170 - Analysis of Algorithms & Data Structures 15 / 16
Analysis of Recursions Substitution method Guess the growth function and prove it using induction. For merge-sort, prove T (n) < Mn log n. This holds for n = 2, n = 3 (base of induction). Fix a value of n and assume the inequality holds for smaller values. we have T (n) = 2T (n/2) + cn 2M(n/2 log n/2) + cn = Mn log n MN + cn Mn log n + cn (the inequality comes from induction hypothesis) Limited Master theorem { a T ( ) n T (n) = b + n c if n > 1 d if n = 1. if log b a > c, then T (n) Θ(n log b a ) if log b a = c then T (n) Θ(n c log n) if log b a < c then T (n) Θ(n c ) COMP 3170 - Analysis of Algorithms & Data Structures 16 / 16