The Growth of Functions A Practical Introduction with as Little Theory as possible
Complexity of Algorithms (1) Before we talk about the growth of functions and the concept of order, let s discuss why we are doing this in the first place. In computer science, we wish to know the complexity of algorithms, i.e. how many operations they require to perform or how much memory they demand. An algorithm is a clearly defined finite sequence of instructions for solving a problem. For example, consider the problem of computing a three-point moving average of a finite sequence of numbers x 1, x 2,, x n. This means that each term in the sequence needs to be replaced by its average with its direct neighbors. This is a common procedure to smooth out data that fluctuates wildly between data points, such as stock market data, or really any measured data that has a lot of noise in it. This problem can be solved by the following algorithm: 1. Replace x 1 by x 1+x 2. 2 2. Go through the numbers k = 2,, n 1 and replace each x k by 3. Replace x n by x n 1+x n. 2 x k 1 + x k + x k+1 3.
Complexity of Algorithms (2) Let us count the number of arithmetic operations (additions and multiplications) required to run the algorithm on the previous slide. Steps 1 and 3 require 2 operations each, and step 2 requires 3 operations times n 2. Therefore, the total number of operations required is 3 n 2 + 4 = 3n 2. Now imagine that we are applying our algorithm to a very large dataset where n could be in the millions. In that case, only the leading term 3n is of practical relevance, and the 2 is negligible. Therefore, for large n, our algorithm essentially requires 3n arithmetic operations. For some purposes, even the constant multiplier 3 is unimportant, and all we need to know is that the number of operations is essentially n times a constant.
Order: a way to describe the scaling behavior of an algorithm (1) Imagine that a financial information system is providing timeaveraged stock market graphs based on a million input points, and we would like to change that number to ten million. Since the number of operations required is essentially n times a constant, we would then need approximately 10 times as much computational power. If the number of operations required was approximately n 2 times a constant, increasing n by a factor 10 would increase computational power required by a factor 100. Instead of saying that 3n 2 is approximately equal to a constant times n for large n, we say that 3n 2 is of order n. This concept of order is a convenient device to express the scaling behavior of an algorithm.
Order: a way to describe the scaling behavior of an algorithm (2) Let s make sure we understand correctly what an order estimate tells us and doesn t tell us about an algorithm. Suppose someone tells us about two algorithms that solve the same problem. A: handles an input of size n in order n operations, and B: handles an input of size n in order n 2 operations. The order estimate only establishes an approximate proportionality for large n. Algorithm A does not necessarily require n operations for an input of size n. It could require 100n + 2 operations, or 10,000n + 500. All of these are order n. All we can say is that A requires approximately Cn operations when n gets large, where C is an unknown constant. Likewise, based on the information given, algorithm B does not (necessarily) require n 2 operations for an input of size n. It could require 0.1n 2 operations, or 10n 2 + 33, or 1,000,000n 2 + 1. All of these are order n 2. All we can say is that B requires approximately Dn 2 operations when n gets large, where D is an unknown constant. For a given n, there is therefore no guarantee that algorithm A will require fewer operations than algorithm B, because C might be large and D might be small. All we know is that A has superior scaling behavior. If we keep increasing n, we will eventually reach a point where A requires fewer operations than B.
A Provisional Definition of Equality of Growth Behavior: Approximate Equality of Ratios Inspired by this example, we make the following provisional definition of order: f(x) is of order g x means that for large x, f x C g(x), where C is a positive constant that does not depend on x. Put differently, two functions f and g grow at the same order if their ratio f x is eventually approximately a constant:, C > 0 for large x. This definition is provisional because approximate equality is not a mathematically rigorous statement, and neither is the condition that x must be large. Limit laws we learned in calculus can help us with order calculations because of the following theorem: If lim g x. f x x g(x) g(x) exists and is a positive number (not zero), then f(x) is of order
Order Calculations based on the Ratio Definition and the Limit Criterion (1) 5x 2 + 7x + 11 is of order x 2, because 5x 2 + 7x + 11 lim x x 2 = 5. The limit law of calculus we applied here is that if two polynomials have the same degree (in this case 2), then their quotient has a limit as x goes to infinity, and the limit is the quotient of the leading coefficients. We were able to omit the absolute values here because the two functions are both positive for sufficiently large x. We use the law here that a polynomial with positive leading coefficient will be positive eventually, i.e. unless it isn t already positive for all x, it will turn, and stay, positive as x grows above some threshold.
Again: Order is about Comparing Growth Rates in Terms of Ratios Let us ponder again that 5x 2 + 7x + 11 is order of x 2. This may seem somewhat counter-intuitive: isn t 5x 2 + 7x + 11 bigger than x 2 and therefore grows faster? It depends on your definition of what grows faster means. If you meant by that that the difference goes to infinity, then you would be correct in saying that 5x 2 + 7x + 11 grows faster than x 2. However, the notion of growth we are developing here is about ratios for large x. For large x, 5x2 +7x+11 x 2 5. In fact, the more x grows, the closer the quotient gets to 5. That s what our limit calculation shows. Therefore, the two quantities are of the same order. A generalization of this result is that any two polynomials of the same degree are of the same order.
Order Calculations Based on the Ratio Definition and the Limit Criterion (2) ln( x 2 + 1) is of order ln x because ln( x 2 + 1) lim x ln x = lim x 2x x 2 + 1 1 x 2x 2 = lim x x 2 + 1 = 2. We used the Rule of L Hospital here to calculate the limit which you should remember from calculus I. Again, we were able to omit the absolute values because both functions are eventually positive (in fact, for x > 0.) Another way to understand this situation is to realize that for large x, x 2 + 1 x 2, so by logarithm laws, ln(x 2 + 1) ln x 2 = 2 ln x.
Order Calculations based on the Ratio Definition and the Limit Criterion (3) Limits can also determine when two functions do not share the same growth rate: f x Theorem: If lim x g(x) For example, 2 x is not of order 3 x because 2 x 2 lim = lim x 3x x 3 is zero or infinity, then f(x) is NOT of order g x. x = 0. Here we have used the calculus law that when you raise a number that is less than 1 in absolute value to powers that go to infinity, the limit is zero. [We were able to omit absolute values because values of exponential functions are always positive.] A generalization of this result is that exponential functions with different bases are not of the same order.
Order Calculations based on the Ratio Definition and the Limit Criterion (4) x 2 3x + 2 is not of order x 3 + 4x 2 + x 1 because lim x x 2 3x + 2 x 3 + 4x 2 + x 1 = 0. Here we have used the calculus law that the limit of a quotient of two polynomials is zero when the numerator polynomial has a lesser degree than the denominator polynomial (2 for the numerator vs 3 for the denominator). We omitted the absolute values because the two polynomials have positive leading coefficients, and are therefore positive eventually.
Orders of Logarithms Any two logarithm functions are of the same order. This is due to the change of base formula which says that two logarithms to a different base are just constant multiples of each other: log b x = log ax log a b for any positive numbers a, b. In particular, this relationship holds if a = e or a = 10: log b x = log 10x log 10 b = ln x ln b Therefore, all the functions log b x are of the same order. This is why sometimes, we use the generic notation log without a base in order-related statements. For example, we might say that the number of computations required by an algorithm with input size n is order of nlog n. The base of the logarithm doesn t matter because the order statement only expresses an approximate proportionality anyway, not an approximate equality. Different interpretations of log n only affect the proportionality constant.
Another look at the absolute values in order-related algebraic manipulations We judge the growth of functions solely based on their absolute values. For two functions f, g to have the same order, we roughly need f x C g(x) for large enough x. The absolute values complicate algebraic manipulation. Fortunately, in many situations, the absolute values can simply be omitted. Since we are only interested in what happens for large enough x, we may start with the assumption that x is positive, or even larger than some suitable positive number. If a polynomial only has non-negative coefficients, then it is non-negative anyway for non-negative inputs. For example, x 3 + 3x + 5 = x 3 + 3x + 5 for x 0. Even when some coefficients are negative, as long as the leading coefficient is positive, we know from calculus that the polynomial will be positive eventually. For example, there must be a real number k such that x 3 5x 2 10x 3 = x 3 5x 2 10x 3 for all x > k. Logarithms to a base greater than 1 are positive if the input is greater than 1: ln x = ln x and log 2 x = log 2 x for x > 1 Exponential functions are always positive anyway: a x and positive a. = a x for all real numbers x
The for order: big-o (1) When the growth rate of f is at most that of g (possibly equal, possibly less), then we say that f is big-o of g. We may also write f x is O(g x ). We will skip the exact definition of that for now and just state how to use limits to make big-o determinations: f x If lim exists and is finite (zero or positive), then f(x) is O g x. x g(x) f x If lim x g(x) =, then f(x) is not O g x. We calculated previously that 2 x lim x 3 x = 0 And concluded that 2 x is not of order 3 x because their ratio does not have a positive limit. However, 2 x is O 3 x because the limit is zero. Generally speaking, if a b, then, and only then, is a x big-o of b x.
The for order: big-o (2) The big-o rules for polynomials are easy: if f and g are polynomials, then f is big-o of g exactly when degree of f degree of g. This follows from the horizontal asymptote rule we know from calculus: if f and g are polynomials, and the degree of f is less than the degree of g, then f x = 0. Is the degrees are equal, then the limit is a positive number. lim x g(x) Either way, f is big-o of g. For example, our previous calculation lim x x 2 3x + 2 x 3 + 4x 2 + x 1 = 0 implies that x 2 3x + 2 is O x 3 + 4x 2 + x 1 because the limit is zero. Therefore, each polynomial is big-o of its own leading power and all higher ones: x 2 + x + 1 is big-o of x 2, of x 3, of x 4, etc.
More Big-O Rules (1) We already said that any polynomial p x is O(x n ) for all n that are greater or equal to the degree of p. Any exponential function with base greater 1 grows faster than any polynomial: p(x) is O(a x ) but a x is not O(p(x)), for all a > 1 and polynomials p x. Logarithms grow slower than all nonconstant polynomials: log x is O(x) but x is not O(log x), no matter what the base of the logarithm. In fact, log x grows so slowly that even when you raise it to a positive power, it is still O x : log x n is O x for any positive n. A consequence of this is that log x n is O p(x) for any nonconstant polynomial p(x).
More Big-O Rules (2) If two functions are big-o of the same g, then so is their sum: (Sum rule) If f 1 (x) and f 2 (x) are both O(g x ), then so is f 1 x + f 2 x. If two terms in a sum have different big-o estimates, then the larger one dominates, and is the big-o estimate for the entire sum: (Extended sum rule) If f 1 (x) is O(g 1 x ) and f 2 (x) is O(g 2 x ), and g 1 is big-o of g 2, then f 1 x + f 2 x is big-o of g 2. If two factors each have a big-o estimate, then the product is big-o of the product of the individual big-o estimates: (Product rule) If f 1 (x) is O(g 1 x ) and f 2 (x) is O(g 2 x ), then f 1 (x) f 2 (x) is O g 1 x g 2 x. Big-O interacts with algebraic inequalities like we would expect it to: Monotonicity rule: if f 1 (x) f 2 (x) and f 2 (x) is O(g x ), then f 1 (x) is O g x as well.
Example of a big-o estimate for a sum of products Let us find the lowest integer n so that f x is O x n, with f x = (x 3 + 1)(log x) 4 +x 4 (1 + x) Solution: we first analyze the two terms separately. Term 1: The factor (x 3 +1) is O x 3. (log x) 4 is O(x). By the product rule, (x 3 +1)(log x) 4 is therefore O x 4. Term 2: The factor x 4 is O x 4. (1 + x) is O(x). By the product rule, x 4 (1 + x) is therefore O x 5. Therefore, f x is O x 5 by the extended sum rule. No lower integer n will do since the second term is not O x 4.
Example of a big-o estimate involving a quotient Let us find the lowest integer n so that f x is O x n, with f x = x4 + x 3 log x x 2 + 1 We use the fact that a quotient gets smaller when the denominator gets bigger to effectively remove the +1 from the denominator:. x 4 + x 3 log x x 2 + 1 < x4 + x 3 log x x 2 = x 2 + x log x. x log x is O x 2. Therefore, by the sum rule, x 2 + x log x is O x 2. By the monotonicity rule, f x is O x 2 as well. Since for large x, the +1 in the denominator is negligible, the inequality above is essentially an equality for large x: f x x 2 + x log x. Since x 2 + x log x is not O(x), n = 2 is the lowest integer such that f x is O x n. [In fact, f is order of x 2.]
Big-O in a nutshell For this summary, let us use the symbol to mean that the left side is big-o of the right side, but not vice versa. (That is not a standard notation. Do not learn it or use it in your own work.) Then: logarithmic polynomial exponential function Lower degree higher degree polynomial. Lower base higher base exponential function.