Big O (Asymptotic Upper Bound) Linear search takes O(n) time. Binary search takes O(lg(n)) time. (lg means log 2 ) Bubble sort takes O(n 2 ) time. n 2 + 2n + 1 O(n 2 ), n 2 + 2n + 1 O(n) Definition: f (n) O(g(n)) iff there exists real c > 0, natural n 0 such that for all natural n n 0, 0 f (n) cg(n) O(g(n)) is the set of such functions. Help! What is it saying?! Let me tell some revisionist history... 1 / 20
How much time does this take? e = false; for (i = 0; i < n; i++) { for (j = i+1; j < n; j++) { if (a[i] == a[j]) { e = true; } } } Platform A Platform B read i, j, n 2 ns 1 ns read a[i] 3 ns 5 ns write i, j 3 ns 1 ns arithmetic 1 ns 1 ns branch 1 ns if continue, 0 ns if continue, 2 ns if exit 2 ns if exit loop back 1 ns 1 ns Total 12n 2 + 14n + 10 10n 2 + 5n + 6 2 / 20
How to be both Coarse and Rigorous? 12n 2 + 14n + 10 10n 2 + 5n + 6 How to unify them and ignore machine differences? What is common about them? Quadratic polynomials. Poster-boy for quadratic polynomials: n 2. How to rigorously say: the above functions are like n 2? And cubic polynomials are like n 3? And 4n lg(n) + 2n + 10 is like n lg(n)? Etc., etc. 3 / 20
How to be both Coarse and Rigorous! Watch this mathemagic: For all natural n 10: 12n 2 + 14n + 10 12n 2 + 14n 2 + 10 = 26n 2 + 10 26n 2 + 10 26n 2 + n 26n 2 + n 2 = 27n 2 There exists natural n 0 such that for all natural n n 0, 0 12n 2 + 14n + 10 27n 2 There exists real c > 0, natural n 0 such that for all natural n n 0, 0 12n 2 + 14n + 10 cn 2 4 / 20
The Rise of Big O There exists real c > 0, natural n 0 such that for all natural n n 0, 0 12n 2 + 14n + 10 cn 2 12n 2 + 14n + 10 O(n 2 ) 10n 2 + 5n + 6 O(n 2 ) My algorithm takes time in O(n 2 ) My algorithm takes O(n 2 ) time My algorithm takes time on the order of n 2 4n lg(n) + 2n + 10 O(n lg(n)) The definition is scary sophisticated because it has to drop some information and carefully preserve some other. It only drops: leading coefficient, small n scenerios, minor terms. 5 / 20
More or Less But wait, these are also true: n O(n 2 ) 3 O(n 2 ) O(n 2 ) includes quadratic functions as well as lesser functions. We need another definition to exclude lesser functions. n O(n 2 ) because it only requires there exists... for all... n cn 2 It s only a one-sided inequality. The definition in the next slide uses a two-sided inequality to rule out this. 6 / 20
Big Θ (Asymptotic Tight Bound) Θ is Greek capital theta. Definition: f (n) Θ(g(n)) iff there exists real b > 0, real c > 0, natural n 0 such that for all natural n n 0, 0 bg(n) f (n) cg(n) Θ(g(n)) is the set of such functions. Example: 12n 2 + 14n + 10 Θ(n 2 ) because for all n 10, 1n 2 12n 2 + 14n + 10 27n 2 Example: n Θ(n 2 ) because (sketch) you can t make bn 2 n to work. 7 / 20
Big Ω (Asymptotic Lower Bound): Big O Flipped Ω is Greek capital omega. Definition: f (n) Ω(g(n)) iff there exists real b > 0, natural n 0 such that for all natural n n 0, 0 bg(n) f (n) Ω(g(n)) is the set of such functions. Equivalently, f (n) Ω(g(n)) iff g(n) O(f (n)). Example: 12n 2 + 14n + 10 Ω(n 2 ) Example: n 2 Ω(n) Example: n Ω(n 2 ) 8 / 20
Asymptotics of Several Variables What if your function takes multiple parameters? I ll illustrate with 2 parameters. f (m, n) O(g(m, n)) iff there exist real c > 0, natural m 0, n 0 such that for all natural m m 0, n n 0, 0 f (m, n) cg(m, n) f (m, n) Θ(g(m, n)) iff there exist real b > 0, real c > 0, natural m 0, n 0 such that for all natural m m 0, n n 0, 0 bg(m, n) f (m, n) cg(m, n) f (m, n) Ω(g(m, n)) iff there exist real b > 0, natural m 0, n 0 such that for all natural m m 0, n n 0, 0 bg(m, n) f (m, n) 9 / 20
Interlude: Functions and Lambda f (n) O(g(n)), n O(n 2 ) is standard but lousy notation. f (n) is not a function, f is. Clever people are not confused by this (they invented this themselves). Enlightened people would not do this to confuse. There is a better way, if you know lambda (Haskell, Scheme, Python, Javascript, Scala, Clojure, F#, C++, Java now too... ): f O(g) iff there exists real c > 0, natural n 0 such that for all natural n n 0, 0 f (n) cg(n) Example: (λn. n + 5) O(λn. n 2 ). Also (λx. x + 5) O(λx. x 2 ). 10 / 20
Using Limits to Prove Big O (Assume: n 0 : n n 0 : f (n) 0 and g(n) 0.) f (n) Theorem: If lim n g(n) exists and is finite, then f (n) O(g(n)). Example: Prove n(n + 1)/2 O(n 2 ) Therefore n(n + 1)/2 O(n 2 ) Example: Prove ln(n) O(n) Therefore ln(n) O(n) n(n + 1)/2 lim n n 2 = 1 2 ln(n) 1/n lim = lim n n n 1 = 0 11 / 20
Using Limits to Disprove Big O (Assume: n 0 : n n 0 : f (n) 0 and g(n) 0.) f (n) Theorem: If lim n g(n) =, then f (n) O(g(n)). Example: Disprove n 2 O(n) Therefore n 2 O(n) Example: Disprove n O(ln(n)) lim n Therefore n O(ln(n)) n 2 lim n n = lim n = n n ln(n) = lim n 1 1/n = lim n n = 12 / 20
When Limits Don t Help f (n) Theorem: If lim n g(n) exists and is finite, then... f (n) Theorem: If lim n g(n) =, then... Which case has not been covered? f (n) If lim n g(n) does not exist and is not, then no conclusion. Hopefully this happens rarely. Example: Define drunk(n) = if n is even then 1 else n drunk(n) O(n) and drunk(n) O(1), but drunk(n) lim n n lim n drunk(n) 1 does not exist and is not does not exist and is not 13 / 20
Using Limits for Θ Theorem: f (n) Θ(g(n)) iff f (n) O(g(n)) and g(n) O(f (n)) Handy when you want to use limits. (Not handy when you want to prove the directly.) Example: n 2 + n 3/2 Θ(n 2 ) n 2 + n 3/2 O(n 2 ) by a limit n 2 O(n 2 + n 3/2 ) by a similar limit Example: ln(n) Θ(n) You saw n O(ln(n)) by a limit. 14 / 20
Big O, Big Θ May Miss Something 10 100 n Θ(n) n + 10 100 Θ(n) Can t say these are practical algorithm times. But O, Θ can t detect them. This is a price for ignoring machine differences. These pathological cases are rare. O and Θ are usually informative. 15 / 20
Big O, Big Θ Gain Simplification e = false; for (i = 0; i < n; i++) { for (j = i+1; j < n; j++) { if (a[i] == a[j]) { e = true; } } } 12n 2 + 14n + 10 ns Took me 30 minutes to figure out (and I got it wrong the first time). The inner loop takes O(n i 1) time, which is O(n). The outer loop takes n iterations. Altogether O(n 2 ) time. This only takes 30 seconds to figure out. It is one of the reasons we use O and Θ. 16 / 20
O vs Θ My friend says: my algorithm takes O(n 2 ) time. It may be actually 9n 2 + 4n + 13. It may be actually 3n + 2. 3n + 2 O(n 2 ) is still true. It may be actually 45. 45 O(n 2 ) is still true. My opinion: You should say Θ as much as you can. Popular opinion: Just say O. A good use of O: Assignment or contract says: Write a O(n 2 )-time algorithm. It allows you to do better. 17 / 20
Worst Case, Best Case e = false; for (i = 0; i < n; i++) { for (j = i+1; j < n; j++) { if (a[i] == a[j]) { e = true; break; } } if (e) { break; } } Depending on what s in a: Anywhere from Θ(1) time to Θ(n 2 ) time. Best case is Θ(1) time. Worst case is Θ(n 2 ) time. We look at worst cases in this course mostly. 18 / 20
Myth Buster Myth: O means worst-case time, Ω means best-case. Truth: O, Ω, Θ classify functions, do not say what the functions stand for. 9n 2 + 4n + 13 may be best-case time, or worst-case time, or best-case space, or worst-case space, or just a polynomial from nowhere. 9n 2 + 4n + 13 O(n 2 ) is true regardless. Best case time is in O(n 2 ) is allowed. It means: Best case time is some function, that function is in O(n 2 ). Clearly a sensible statement and possible scenerio. O, Ω, Θ are good for any function from natural to non-negative real. 19 / 20
Assumptions on Computational Time This course assumes the following to take O(1) time each: 64-bit integer arithmetic; 64-bit floating-point arithmetic; 64-bit pointer comparison read/write 64 bits (data or pointer), if the address is also 64-bit (note: this limits memory size) branching, jumping Food for thought: What if your numbers have many digits and exceed 64 bits? What if you want/have unlimited memory? I run mergesort, but only on 2GB arrays ever, so O(n lg n) time but n 2 31 ever. Is this O(1) time? 20 / 20