The Growth of Fuctios Theoretical Supplemet
The Triagle Iequality The triagle iequality is a algebraic tool that is ofte useful i maipulatig absolute values of fuctios. The triagle iequality says that the absolute value of the sum of two real umbers is o more tha the sum of their absolute values: a + b a + b The iequality is i fact a equality if a ad b have the same sig. If they have opposite sigs, the there is partial or full cacellatio o the left side, which is the less tha the right side. It is ot difficult to tur this observatio ito a exact proof. The triagle iequality is valid for more tha two terms: k=1 a k There is also the reverse triagle iequality: a + b a b for all real umbers a, b. k=1 a k
Towards a Exact Defiitio of Order Let us thik about how to come up with a exact defiitio of order. f(x) beig of order g x defies a very limited sort of equality betwee f(x) ad g x, amely equality i terms of their log-term growth behavior. For real umbers, equality ca be expressed i terms of iequalities: a = b iff a b ad b a. Ispired by this aalogy, we will defie what is essetially a less tha or equal relatio for order, ad a greater tha or equal relatio for order, ad the use that to defie the equal relatioship.
The Less Tha or Equal Relatio for Growth of Fuctios: Big-O Defiitio: f(x) is O(g x ) ( big-o of g x ) meas that there is some costat C ad some costat k such that f x C g(x) for all x > k. No-techically speakig, f(x) beig O(g x ) meas that the log-term growth of f is o faster tha that of g. Examples: 5x 2 + 3x is O(x 2 ) 5x 2 + 3x is O x 3 5x 2 + 3x is O(x ) for all 2. Ay polyomial p x is O(x ) if is at least the degree of p. The pair of values (C, k) is ot uique. Give a C, if oe k works, the ay larger k works too. Icreasig C usually has the effect that lower values will work for k. Rule of thumb: Larger C correspods to smaller k.
Ca we chage the iequalities i the defiitio of big-o from strict to ostrict ad vice versa? The official defiitio we gave is that f(x) is O(g x ) iff that there is some costat C ad some costat k such that f x C g(x) for all x > k. Observe that if f x < C g(x) is true for some x, the f x C g(x) is also true. Furthermore, if somethig is true for all x k, the it is also true for all x > k. Therefore, to show that f(x) is O g x suitable costats C ad k is sufficiet:, ay of the followig three coditios for 1. f x C g(x) for all x k, 2. f x < C g(x) for all x > k, 3. f x < C g(x) for all x k. Ofte, showig oe of these three is more atural or coveiet tha showig the origial coditio.
Visualizig a Big-O Relatioship Let us prove that 4x + 5 is O x 2. Ituitively, this makes immediate sese: a liear fuctio grows less quickly tha a quadratic oe. Let s make this precise with the help of a graph. Sice quadratic growth beats liear growth, we kow that 4x + 5 < x 2 evetually. By graphig, we see that while 4x + 5 is above x 2 for a bit ear the origi, it is less tha x 2 after x = 5 (we ca cofirm this algebraically): 4x + 5 < x 2 for x > 5. Absolute values have o effect o either fuctio for x > 0, much less for x > 5, therefore 4x + 5 < x 2 for x > 5. Thus, 4x + 5 < C x 2 O x 2. for x > k where C = 1 ad k = 5. Therefore, 4x + 5 is
Provig a Big-O Relatioship (1) Based o what we have already leared, 5x 2 + 3x is order of x 2. Therefore, 5x 2 + 3x is also O(x 2 ). We have already discussed the oly apparet cotradictio i that statemet. We re ot sayig that 5x 2 + 3x < x 2. We re sayig that 5x 2 + 3x < Cx 2 evetually, for a suitable positive costat C. What might that costat be? C = 5 wo t work because 5x 2 + 3x is ot less tha 5x 2 if x is positive. Ay C > 5 should work though, because ituitively, eve a little more quadratic is goig to beat the liear term 3x. For coveiece, we try C = 6, i.e. we try to prove that 5x 2 + 3x < 6x 2 evetually. That iequality is equivalet to 3x < x 2. Sice we are oly iterested i makig this true for large x, we may assume x > 0, which permits us to divide 3x < x 2 by x ad get 3 < x (which is ormally a very bad idea, dividig (i)equalities by variable quatities). Therefore, 5x 2 + 3x < 6x 2 for x > 3. Agai, absolute values have o effect o either fuctio for x > 0, much less for x > 3, therefore 5x 2 + 3x < 6x 2 for x > 3. Thus, 5x 2 + 3x < C x 2 O x 2. for x > k where C = 6 ad k = 3. Therefore, 5x 2 + 3x is
Provig a Big-O Relatioship (2) Let us prove agai that 5x 2 + 3x is O x 2, but this time, let s attempt a more mechaical approach that substitutes symbolic maipulatio for reasoig about the two fuctios. It is a algebra fact that for x > 1, x < x 2. Therefore, for x > 1, we have 5x 2 + 3x < 5x 2 + 3x 2 = 8x 2. Therefore, if we let k = 1 ad C = 8, we have f x < Cg(x) for all x > k. Agai, we ca just itroduce absolute values here without chagig the fuctios ivolved: 5x 2 + 3x < 8 x 2 for x > 1 That meas we have verified the defiitio of 5x 2 +3x beig O x 2 : We have show that for k = 1 ad C = 8, 5x 2 +3x < C x 2 for all x > k. Notice agai that the pair (C, k) is ot uique ad that the larger C correspods to a smaller k. The techique show here exploits the fact that for x > 1, a higher power of x is bigger tha a smaller power of x. This way, we are able to come up with a upper boud for the polyomial f simply by replacig each lower power by the highest power.
Provig a Big-O Relatioship (3) Let us cosider whether 5x l x is O x 2. Graphig both fuctios substatiates this. Apparetly, 5x l x x 2 for x > 13. Of course, eye-ballig such iformatio based o a graph is ot proof. We kow already that the logarithmic growth of l x is weaker tha the growth of x. Therefore, it stads to reaso that 5x l x grows less tha 5x 2. Thus, we suspect for sufficietly large x. 5x l x 5x 2 Therefore, we wish to prove that l x x for sufficietly large x. There is o algebraic way to solve mixed logarithmic-polyomial (i)equalities. However, we ca prove such a iequality usig the methods of calculus.
Iterlude: Usig Calculus where Algebra Fails Observe that the derivatives of the two quatities l x ad x are very simple fuctios: 1 ad 1. We ca easily determie a relatioship betwee them: if x x 1, the 1 1. Havig established a iequality betwee the derivatives, x we ca get a iequality betwee the fuctios through defiite itegratio usig a coveiet lower limit ad a variable upper limit: න 1 x 1 t dt න 1 Evaluatig the itegrals, we get l x x 1 < x for x 1. Havig proved this, we ow also kow for sure that 5x l x < 5x 2 for x 1. [Observe that we eded up provig that 5x l x is big-o of x 2 with C = 5 ad k = 1, ot C = 1 ad k = 13 as graphig first suggested. Oce agai, the larger C value correspods to a lower k value.] x dt
How big-o estimates iteract with products Just earlier, we used the followig reasoig: We kow already that the logarithmic growth of l x is weaker tha the growth of x. Therefore, it stads to reaso that 5x l x grows less tha 5x 2. Thus, we suspect 5x l x 5x 2 for sufficietly large x. This is a applicatio of the followig product rule': if : f 1 (x) is O(g 1 x ) ad f 2 (x) is O(g 2 x ), the f 1 (x) f 2 (x) is O g 1 x g 2 x. I words: give two fuctios, each with its ow big-o estimate, the the product of the two big-o estimates is a big-o estimate for the product of the fuctios. We prove this based o the defiitio of the big-o relatioship. If f 1 x C 1 g 1 (x) for all x > k 1, ad f 2 x C 2 g 2 (x) for all x > k 2, the f 1 x f 2 (x) C 1 C 2 g 1 x g 2 (x) for all x > max{ k 1, k 2 }. Thus, we showed that f 1 (x) f 2 (x) is O g 1 x g 2 x with C = C 1 C 2 ad k = max{ k 1, k 2 }. Example: sice l x is O(x), x 3 l x is O x 4.
How big-o estimates iteract with sums There is a sum rule as well: if two fuctios f 1 ad f 2 are both big-o of the same fuctio g, the so is their sum. We prove this directly based o the defiitio of big-o: If f 1 x C 1 g(x) for all x > k 1, ad f 2 x C 2 g(x) for all x > k 2, the f 1 x + f 2 (x) f 1 x + f 2 (x) (C 1 +C 2 ) g x for all x > max{ k 1, k 2 }. Thus, we showed that f 1 x + f 2 (x) is O g(x) with C = C 1 + C 2 ad k = max{ k 1, k 2 }. Example 1: x 2 l x ad 5x 3 are both O x 3. Therefore, x 2 l x + 5x 3 is also O x 3. Example 2: x 4 l x is O x 5. 3x 3 is O x 3, but also O x 5. Therefore, x 4 l x + 3x 3 is O x 5. (Remark: x 4 l x + 3x 3 is also O x 4 l x, a slightly better big-o estimate.) Example 2 illustrates a extesio of the sum rule we discussed above: If two terms i a sum have differet big-o estimates, the the larger oe domiates, ad is the big-o estimate for the etire sum.
A more complex example of a big-o estimate for a sum of products Let us fid the lowest iteger so that is O x. f x = (x 3 + 1)(log x) 4 +x 4 (1 + x) Solutio: we first aalyze the two terms separately. Term 1: The factor (x 3 +1) is O x 3. (log x) 4 is O(x). By the product rule, (x 3 +1)(log x) 4 is therefore O x 4. Term 2: The factor x 4 is O x 4. (1 + x) is O(x). By the product rule, x 4 (1 + x) is therefore O x 5. Therefore, f x is O x 5. (No lower iteger will do sice the secod term is ot O x 4.)
Big-O Estimates for sums where the umber of terms icreases to ifiity (1) Let us cosider the fuctio f = k = 1 + 2 + +. k=1 Based o what we leared about big-o estimates for sums, we could make the followig icorrect argumet: each term i the sum is at most, therefore O. If each term is O, the so is the sum. Thus f is O. This reasoig is icorrect because the umber of terms itself is a variable here- there are of them. Sice each term is at most, ad there are of them, the the sum is at most = 2. That is ot a costat times. You ca apply the sum rule for big-o estimates to sums that have more tha two terms, but the umber of terms must still be costat or at least be bouded (o bigger tha some fixed maximum). The umber of terms caot go to ifiity if you wish to apply that rule. O the ext page, we discuss ways of fidig correct big-o estimates for sums like the oe above.
Big-O Estimates for sums where the umber of terms icreases to ifiity (2) We have earlier leared the summatio formula k = 1 + 2 + + = k=1 This shows that 1 + 2 + + is big-o of 2. ( + 1) 2 We could have derived this relatioship without kowledge of the summatio formula for cosecutive itegers by oticig that each term i the sum is less tha : 1 + 2 + + < + + + = = 2 This is a useful approach for gettig big-o estimates of other sums. Examples: 1 2 + 2 2 + + 2 < 2 + 2 + + 2 = 2 = 3 Therefore, 1 2 + 2 2 + + 2 is big-o of 3. Likewise, 1 3 + 2 3 + + 3 is big-o of 4, etc. Usig summatio formulas, we ca show that these big-o estimates are optimal: 1 p + 2 p + + p is order of (p+1) for positive itegers p..
The Greater Tha or Equal Relatio for Order: Big-Omega Defiitio: f(x) is Ω(g x ) ( big-omega of g x ) meas that there are positive costats C ad k such that f x C g(x) for all x > k. No-techically speakig f(x) beig Ω(g x ) meas that the log-term growth of f is at least as fast as that of g. Examples: 5x 2 + 3x is Ω(x 2 ) 5x 2 + 3x is Ω(x 1 ) 5x 2 + 3x is Ω(x ) for all 2.
Provig a Big-Omega Relatioship Let us prove that f x = 5x 2 + 3x is Ω g x where g x = x 2. For x > 1, 3x > 0. Therefore, for x > 1, we have 5x 2 + 3x 5x 2. Therefore, if we let k = 1 ad C = 5, we have f x Cg(x) for all x > k. Sice f x ad g(x) are o-egative for x > 1, we may itroduce absolute values: 5x 2 + 3x 5 x 2 for x > 1 That meas we have verified the defiitio of f(x) beig Ω g(x) : We have show that there exist costats k ad C such that f x C g(x) for all x > k. By re-usig the relatioship betwee differet powers of x for x > 1, we also get Therefore, f(x) is Ω x for all 2. 5x 2 + 3x 5x for ay 2.
A Rigorous Defiitio of Order Defiitio: f(x) is of order g x, or Θ(g x ) ( big- Theta of g x ) meas that f(x) is O(g x ) ad f(x) is Ω(g x ). Example: we demostrated that f x = 5x 2 + 3x is O x for all 2 ad is Ω(x ) for all 2. Therefore, there is oe value of (ad oly oe such value) for which f(x) is both. That shows: 5x 2 + 3x is Θ x 2.
The order of polyomials We already metioed that a th degree polyomial is always of order x. We are ow i a positio to give a proof of that theorem. Suppose the th degree polyomial is f x = where a 0. a k x k = a x + a 1 x 1 + + a 0 We will show that f x is O(x ) ad that f x is Ω x
f x is O(x ) Let us assume x > 1. Sice we kow othig about the sigs of the coefficiets of our geeric polyomial f x, we must use the triagle iequality: f x = a k x k a k x k = a k x k We ow use the fact that for x > 1, x k x for all k. Therefore, If we ow defie f x a k x k = x a k C = a k the we have f x C x for all x > 1, which proves that O(x ).
f x is Ω x For this part, we shall rewrite f x by factorig out the leadig term such as i the followig example: 2x 2 + 3x + 1 = 2x 2 1 + 3 2x + 1 2x 2 The calculatio looks like this i geeral: a k x k = a x ak a x k = a x The secod factor o the right ca be broke up as follows: 1 a k a x k = a k a x k + 1 0 as x a k a x k
f x is Ω x Accordig to calculus, the sum 1 a k a x k goes to zero as x because every term is a costat times a reciprocal power of x. Therefore, by defiitio of limit, that term will evetually (for sufficietly large x) be less tha 1 i absolute value. Therefore, we ca fid k > 0 so that for all x > k, 2 1 Therefore, by the reverse triagle iequality, a k a x k < 1 2 1 1 + a k a x k 1 1 a k a x k
f x is Ω x O the right side of the iequality, 1 1 + a k a x k 1 1 a k a x k the outer absolute value ca be omitted sice the ier absolute value is at most ½, so subtractio of that from oe caot result i a egative umber. The differece has to be at least ½.Therefore, 1 1 + a k a x k > 1 2
f x is Θ x Now the heavy liftig is doe. We have show that there is some positive k so that for all x > k, f(x) = a x 1 Therefore, recogizig the a Ω x. a k a x k + 1 > a 2 x 2 as the costat C, f(x) is Havig already proved that f x is O(x ), we have therefore proved that f x is Θ x.