Introduction to Algorithms: Asymptotic Notation

Why Should We Care? (Order of growth) CS 421 - Analysis of Algorithms 2

Asymptotic Notation O-notation (upper bounds): that 0 f(n) cg(n) for all n n. 0 CS 421 - Analysis of Algorithms 3

Asymptotic Notation O-notation (upper bounds): that 0 f(n) cg(n) for all n n. 0 CS 421 - Analysis of Algorithms 4

Asymptotic Notation O-notation (upper bounds): that 0 f(n) cg(n) for all n n. 0 where c = 1, n 0 = 2 CS 421 - Analysis of Algorithms 5

Asymptotic Notation O-notation (upper bounds): that 0 f(n) cg(n) for all n n. 0 functions, not values CS 421 - Analysis of Algorithms where c = 1, n 0 = 2 funny, one-way equality 6

Asymptotic Notation O-notation (upper bounds): that 0 f(n) cg(n) for all n n. 0

Set Definition of Ο-notation Ο(g n ) is the set of all functions with a SMALLER or the SAME order of growth as g n. CS 421 - Analysis of Algorithms 9

Set Definition of Ο-notation Ο(g n ) is the set of all functions with a SMALLER or the SAME order of growth as f(n). EXAMPLE: Ο(n ' ) = 5n ', 10n - + 500, logn, Or put another way: 5n ' Ο(n ' ) CS 421 - Analysis of Algorithms 10

Ω-notation (lower bounds)

Asymptotic Notation Ω-notation (lower bounds): that 0 f(n) cg(n) for all n n. 0 CS 421 - Analysis of Algorithms 12

Asymptotic Notation Ω-notation (lower bounds): that 0 f(n) cg(n) for all n n. 0 CS 421 - Analysis of Algorithms 13

Asymptotic Notation Ω-notation (lower bounds): that 0 f(n) cg(n) for all n n. 0 where c = 1, n 0 = 16 CS 421 - Analysis of Algorithms 14

Asymptotic Notation Ω-notation (lower bounds): that 0 f(n) cg(n) for all n n. 0 CS 421 - Analysis of Algorithms 15

Set Definition of Ω-notation Ω(g n ) is the set of all functions with a LARGER or the SAME order of growth as g n. EXAMPLE: Ω(n - ) = 5n -, 10n ' + 500, n - log n, Or put another way: 5n - Ω(n - ) CS 421 - Analysis of Algorithms 16

Θ-notation (tight bounds) Combine definitions of Ο and Ω: We write: f n = Θ g n if there exist constants c 1 > 0, c 2 > 0 and n 0 > 0 such that c 1 g n <= f n <= c 2 g(n) CS 421 - Analysis of Algorithms 17

Θ-notation (tight bounds) CS 421 - Analysis of Algorithms 19

Θ-notation (tight bounds) CS 421 - Analysis of Algorithms 20

Θ-notation (tight bounds) Upper bound CS 421 - Analysis of Algorithms 21

Θ-notation (tight bounds) Upper bound Lower bound CS 421 - Analysis of Algorithms 22

Θ-notation (tight bounds) Upper bound Lower bound CS 421 - Analysis of Algorithms 23

Θ-notation (tight bounds) EXAMPLE: CS 421 - Analysis of Algorithms 24

O-notation like. o-notation like <. ο-notation We write: f n = ο g n for any constants c > 0, there is a constant n M > 0 such that 0 < f(n) < cg(n) CS 421 - Analysis of Algorithms 25

O-notation like. o-notation like <. ο-notation We write: f n = ο g n for any constants c > 0, there is a constant n M > 0 such that 0 < f(n) < cg(n) EXAMPLE: 2n 2 = o(n 3 ), n 0 = 2/c CS 421 - Analysis of Algorithms 26

Set Definition of ο-notation ο(g n ) is the set of all functions with a strictly SMALLER order of growth as g n. CS 421 - Analysis of Algorithms 27

Set Definition of ο-notation ο(g n ) is the set of all functions with a SMALLER order of growth as g n. EXAMPLE: ο(n ' ) = 10n - + 500, log n, Or put another way: 10n - + 500 ο(n ' ) CS 421 - Analysis of Algorithms 28

Ω-notation is like. ω-notation is like >. ω-notation We write: f n = ω g n for any constants c > 0, there is a constant n M > 0 such that 0 < cg(n) < f(n) CS 421 - Analysis of Algorithms 29

Ω-notation is like. ω-notation is like >. ω-notation We write: f n = ω g n for any constants c > 0, there is a constant n M > 0 such that 0 < cg(n) < f(n) EXAMPLE: n = ω(lg n), CS 421 - Analysis of Algorithms n 0 = 1+1/c 30

Set Definition of ω-notation ω(f(n)) is the set of all functions with a strictly LARGER order of growth as f(n). CS 421 - Analysis of Algorithms 31

Set Definition of ω-notation ω(f(n)) is the set of all functions with a strictly LARGER order of growth as f(n). EXAMPLE: ω(n - ) = 10n ' + 500, n - logn, Or put another way: 10n ' ω(n - ) CS 421 - Analysis of Algorithms 32

Properties: Asymptotic Notation Transitivity Reflexivity Symmetry Transposition CS 421 - Analysis of Algorithms 33

Transitivity Assuming f(n) and g(n) are asymptotically positive: f(n) = Θ(g(n)) and g(n) = Θ(h(n)) implies f(n) = Θ(h(n)) Holds for Ο, Ω, ο, and ω relations as well. CS 421 - Analysis of Algorithms 34

Reflexivity Assuming f(n) is asymptotically positive: f(n) = Ο(f(n)) and f(n) = Ω(f(n)) and f(n) = Θ(f(n)) DOES NOT hold for ο and ω relations. CS 421 - Analysis of Algorithms 35

Symmetry Assuming f(n) and g(n) are asymptotically positive: f(n) = Θ(g(n)) iff (if, and only if,) g(n) = Θ(f(n)) CS 421 - Analysis of Algorithms 36

Transpose Assuming f(n) and g(n) are asymptotically positive: f(n) = Ο(g(n)) iff g(n) = Ω(f(n)) and f(n) = ο(g(n)) iff g(n) = ω(f(n)) CS 421 - Analysis of Algorithms 37

Analogy with numbers CS 421 - Analysis of Algorithms 38

Analogy with numbers CS 421 - Analysis of Algorithms 39

Analogy with numbers CS 421 - Analysis of Algorithms 40

Useful Property f1(n) = Θ(g1(n)) and f2 n = Θ(g2(n)) implies f1 n + f2 n = Θ(max {g1(n), g2(n)}) Holds for Ο, Ω, ο, and ω relations as well. CS 421 - Analysis of Algorithms 41

Using Limits to Compare Growth Rates Though using Ο, Ω, ο, and ω indispensable for comparing growth rates of functions in the abstract, when comparing actual functions, convenient to CS 421 - Analysis of Algorithms 42

Using Limits to Compare Growth Rates lim V X f(n) g(n) = 0 - f(n) has smaller growth rate than g(n) c - f(n) has same growth rate as g(n) - f(n) has larger growth rate than g(n) first two cases f n Ο g n last two cases f n Ω(g n ) second case f n Θ(g n ) CS 421 - Analysis of Algorithms 43

Growth Rates (Example 1) CS 421 - Analysis of Algorithms 44

Growth Rates (Example 1) CS 421 - Analysis of Algorithms 45

Growth Rates (Example 1) CS 421 - Analysis of Algorithms 46

Growth Rates (Example 2) CS 421 - Analysis of Algorithms 47

Growth Rates (Example 2) CS 421 - Analysis of Algorithms 48

Growth Rates (Example 2) L'Hôpital's rule https://en.wikipedia.org/wiki/l%27hôpital%27s_rule CS 421 - Analysis of Algorithms 49

Growth Rates (Example 2) L'Hôpital's rule https://en.wikipedia.org/wiki/l%27hôpital%27s_rule CS 421 - Analysis of Algorithms 50

Growth Rates (Example 2) Derivatives https://www.wyzant.com/resources/lessons/m ath/calculus/differentiation/list_of_derivatives http://www.ripmat.it/mate/c/cf/cfdb.html CS 421 - Analysis of Algorithms 51

Growth Rates (Example 2) CS 421 - Analysis of Algorithms 52

Growth Rates (Example 3) CS 421 - Analysis of Algorithms 53

Growth Rates (Example 3) Stirling's approximation https://en.wikipedia.org/wiki/stirling%27s_approximation CS 421 - Analysis of Algorithms 54

Growth Rates (Example 3) CS 421 - Analysis of Algorithms 55

Growth Rates (Example 3) CS 421 - Analysis of Algorithms 56

Most Common Growth Rates Class Name Examples 1 Constant Only used in best-case efficiencies. log n logarithmic Result of cutting problem size by a constant factor, like Binary Search n linear Algorithms that scan a list of size n, like Sequential, or Linear, Search n log n n-log-n Divide-and-Conquer algorithms, like Merge Sort and Quick Sort n2 quadratic Efficiencies with two embedded loops, Bubble Sort and Insertion Sort n3 cubic Efficiencies with three embedded loops, like many linear algebra algorithms 2n exponential Algorithms that generate all sub-sets of an n-element set n! factorial Algorithms that generate all permutations CS 421 - Analysis of an of Algorithms n-element set 57

Macro Substitution Convention: A set in a formula represents an anonymous function in the set.

Macro substitution Convention: A set in a formula represents an anonymous function in the set. EXAMPLE: f(n) = n 3 + O(n 2 ) means f(n) = n 3 + h(n) for some h(n) O(n 2 ) CS 421 - Analysis of Algorithms 59

Macro substitution Convention: A set in a formula represents an anonymous function in the set. EXAMPLE: CS 421 - Analysis of Algorithms 60