Compare the growth rate of functions

Compare the growth rate of functions We have two algorithms A 1 and A 2 for solving the same problem, with runtime functions T 1 (n) and T 2 (n), respectively. Which algorithm is more efficient? We compare the growth rate of T 1 (n) and T 2 (n). If T 1 (n) = Θ(T 2 (n)), then the efficiency of the two algorithms are about the same (when n is large). If T 1 (n) = o(t 2 (n)), then the efficiency of the algorithm A 1 will be better than that of algorithm A 2 (when n is large). By using the definitions, we can directly show whether T 1 (n) = O(T 2 (n)), or T 1 (n) = Ω(T 2 (n)). However, it is not easy to prove the relationship of two functions in this way. c Jinhui Xu (University at Buffalo) CSE 431/531 Algorithm Analysis and Design 2 / 36

Limit Test Limit Test is a powerful method for comparing functions. Limit Test Let T 1 (n) and T 2 (n) be two functions. Let c = lim n T 1(n) T 2(n). 1 If c is a constant > 0, then T 1 (n) = Θ(T 2 (n)). 2 If c = 0, then T 1 (n) = o(t 2 (n)). 3 If c =, then T 1 (n) = ω(t 2 (n)). 4 If c does not exists (or if we do not know how to compute c), the limit test fails. T Proof of (1): c = lim 1(n) n T means: ɛ > 0, there exists n 2(n) 0 0 such that for any n n 0 : T1(n) T c 2(n) ɛ; or equivalently: c ɛ T 1(n) T 2(n) c + ɛ. Let ɛ = c/2 and let c 1 = c ɛ = c/2 and c 2 = c + ɛ = 3c/2, we have c 1 T 2 (n) T 1 (n) c 2 T 2 (n) for all n n 0. Thus T 1 (n) = Θ(T 2 (n)) by definition. c Jinhui Xu (University at Buffalo) CSE 431/531 Algorithm Analysis and Design 4 / 36

Example Example 1 T 1 (n) = 10n 2 + 15n 60, T 2 (n) = n 2 T 1 (n) lim n T 2 (n) = lim 10n 2 + 15n 60 n n 2 = lim (10+15 n n 60 ) = 10+0 0 = 10 n2 Since 10 is a constant > 0, we have T 1 (n) = Θ(T 2 (n)) = Θ(n 2 ) by the statement 1 of Limit Test (as expected). c Jinhui Xu (University at Buffalo) CSE 431/531 Algorithm Analysis and Design 5 / 36

Log function The log functions are very useful in algorithm analysis. lg = log 2 n log n = log 10 n ln n = log e n (ln n is the log function with the natural base e = 2.71828...). c Jinhui Xu (University at Buffalo) CSE 431/531 Algorithm Analysis and Design 6 / 36

Log base change formula Log base change formula For any 1 < a, b, log b n = log a n log a b = log b a log a n. Proof: Let k = log b n. By definition: n = b k. Take log a on both sides: log a n = log a (b k ) = k log a b This implies: log b n = k = log a n log a b. Let n = a in this formula and note 1 = log a a: log b a = log a a log a b = 1 log a b This proves the second part of the formula. c Jinhui Xu (University at Buffalo) CSE 431/531 Algorithm Analysis and Design 7 / 36

L Hospital Rule L Hospital Rule If lim n f (n) = 0 and lim n g(n) = 0, then f (n) lim n g(n) = lim f (n) n g (n) If lim n f (n) = and lim n g(n) =, then f (n) lim n g(n) = lim f (n) n g (n) c Jinhui Xu (University at Buffalo) CSE 431/531 Algorithm Analysis and Design 9 / 36

Example Example 2 T 1 (n) = n 2 + 6, T 2 (n) = n lg n. (Recall: lg n = log 2 n.) T lim 1 (n) n T 2 (n) = lim n n2 +6 n lg n = lim n+ 6 n n lg n 1 6 n = lim 2 n 1 (by L Hospital Rule) ln 2 n = ln 2 lim n (n 6 n ) = ln 2( 0) = By Limit Test, we have n 2 + 6 = ω(n lg n). c Jinhui Xu (University at Buffalo) CSE 431/531 Algorithm Analysis and Design 10 / 36

Example Example 3 T 1 (n) = (ln n) k, T 2 (n) = n ɛ, where k > 0 is any (large) constant and ɛ > 0 is any (small) constant. (Recall: ln n = log e n.) T lim 1 (n) n T 2 (n) = lim (ln n)k n n ɛ k(ln n) = lim k 1 (1/n) n = k ɛ lim n = k(k 1) ɛ 2 ɛn (ɛ 1) (ln n)k 1 n ɛ lim n (ln n) k 2 n ɛ... = k(k 1) 2 1 ɛ k lim n 1 n ɛ = 0 (use L Hospital Rule) (use L Hospital Rule again and simplify) (use L Hospital Rule k times) So by Limit Test, (ln n) k = o(n ɛ ) for any k and ɛ. For example, take k = 100 and ɛ = 0.01, we have (ln n) 100 = o(n 0.01 ). c Jinhui Xu (University at Buffalo) CSE 431/531 Algorithm Analysis and Design 11 / 36

Example Example 4 T 1 (n) = n k, T 2 (n) = a n, where k > 0 is any (large) constant and a > 1 is any constant bigger than 1. lim n T 1 (n) T 2 (n) = lim n nk a n = lim n k n k 1 ln a a n = k = k(k 1) 2 1 (ln a) k lim n n 0 a n ln a lim n nk 1 a n = k(k 1) 2 1 (ln a) k lim n 1 a n = 0 (using L Hospital Rule) ( using L Hospital Rule k times) So by Limit Test, n k = o(a n ) for any k > 0 and a > 1. For example, take k = 1000 and a = 1.001, we have n 1000 = o((1.001) n ). c Jinhui Xu (University at Buffalo) CSE 431/531 Algorithm Analysis and Design 12 / 36

Example Example 5 T 1 (n) = log a n, T 2 (n) = log b n, where a > 1 and b > 1 are any two constants bigger than 1. By the Log Base Change Formula: log b n = log b a log a n T Thus: lim 1 (n) n T 2 (n) = lim n log a n log b n = lim n log a n log b a log a n = 1 log b a Since 1 log b a > 0 is a constant, we have log a n = Θ(log b n) by Limit Test. So: the growth rates of the log functions are the same for any base > 1. c Jinhui Xu (University at Buffalo) CSE 431/531 Algorithm Analysis and Design 13 / 36

Example Example 6 T 1 (n) = a n, T 2 (n) = b n, where 1 < a < b are any two constants. We have: lim n T 1 (n) T 2 (n) = lim n an b n = lim n ( a b )n = 0. Thus: a n = o(b n ) (for any 1 < a < b) by Limit Test. The list of common functions: The following list shows the functions commonly used in algorithm analysis, in the order of increasing growth rate (a, b, c, d, k, ɛ are positive constants, ɛ < 1, k > 1, d > 1 and a < b): c, log d n, (log d n) k, n ɛ, n, n k, a n, b n, n!, n n in the sense that if f (n) and g(n) are any two consecutive functions in the list, we have f (n) = o(g(n)) c Jinhui Xu (University at Buffalo) CSE 431/531 Algorithm Analysis and Design 14 / 36

Examples Example 7 T 1 (n) = n! and T 2 (n) = a n (a > 1) lim n a n n! =? L Hospital Rule doesn t help: We don t know how to take derivative of n! n! = a 1 a 2 a a 2 a 2 a + 1 a n }{{}}{{} 2 a terms (n 2 a ) terms a n The first part is a constant c > 0. In the second part, each term < 1/2. So: a n 0 lim n n! c lim (1 n 2 )(n 2 a ) = 0 By Limit Test: a n = o(n!). c Jinhui Xu (University at Buffalo) CSE 431/531 Algorithm Analysis and Design 15 / 36

Stirling Formula Stirling Formula ( n ) n 1 ( 2πn e 12n+1 n ) n 1 n! 2πn e 12n e e or: n! = ( n ) ( ( )) n 1 2πn 1 + Θ e n When n = 10; n! = 3628800. ( 2πn n ) n e = 3598696, 99% accurate. c Jinhui Xu (University at Buffalo) CSE 431/531 Algorithm Analysis and Design 17 / 36

Examples Example 7 (another solution) T 1 (n) = n! and T 2 (n) = a n (a > 1) n! lim n a n lim ( n ) n ( n ) n 2πn = lim 2πn lim n ae n n ae The first limit is. The second limit goes to. So it s also. Thus n! lim n a = and n! = ω(a n ) by limit test. n Example 8 T 1 (n) = n n and T 2 (n) = n! By using similar method as in Example 7, we can show: n! = o(n n ) c Jinhui Xu (University at Buffalo) CSE 431/531 Algorithm Analysis and Design 18 / 36

Summations Consider the following simple program: 1: for i = 1 to n do 2: the loop body takes Θ(i k ) time 3: end for What s the runtime of this program? It should be: T(n) = n n Θ(i k ) = c i k (for some constant c) i=1 i=1 Thus, it is important to know the sum of the form n i=1 ik. c Jinhui Xu (University at Buffalo) CSE 431/531 Algorithm Analysis and Design 20 / 36

Summation formulas n i 1 = 1 + 2 + n = i=1 n i 2 = 1 2 + 2 2 + n 2 = i=1 i=1 n(n + 1) 2 = Θ(n 2 ) (1) n(n + 1)(2n + 1) 6 = Θ(n 3 ) (2) n i 3 = 1 3 + 2 3 + n 3 = n2 (n + 1) 2 = Θ(n 4 ) (3) 4 In general, for any k > 0, the following is true. n i k = Θ(n k+1 ) (4) i=1 c Jinhui Xu (University at Buffalo) CSE 431/531 Algorithm Analysis and Design 21 / 36

Summations: By using these formulas, we can compute the runtime of nested loops. Example for i = 1 to n do for j = i to n do for k = i to j do (... loop body takes Θ(1) time.) end for end for end for Since the inner loop body takes Θ(1) time, we only need to count the number D(n) of the inner loop iterations. Then T(n) = D(n) Θ(1) = Θ(D(n)). D(n) = n n j i=1 j=i k=i 1 = n n i=1 j=i (j i + 1) c Jinhui Xu (University at Buffalo) CSE 431/531 Algorithm Analysis and Design 22 / 36

Calculate D(n) To calculate the second sum, let t = j i + 1. When j = i, t = 1. When j = n, t = n i + 1. Thus n (j i + 1) = j=i n i+1 t=1 t = 1 + 2 + (n i + 1) = (n i + 2)(n i + 1) 2 Next we calculate: n (n i+2)(n i+1) i=1 2. Let s = n i + 1. When i = 1, s = n. When i = n, s = 1. Thus: = n s=1 (s+1)s 2 = 1 2 { n s=1 s2 + n s=1 s} = 1 2 { n(n+1)(2n+1) 6 + n(n+1) 2 } = Θ(n 3 ) c Jinhui Xu (University at Buffalo) CSE 431/531 Algorithm Analysis and Design 23 / 36

More Summations: The following summation formulas are useful. 1 a n n+1 a i = 1 + a + a 2 + + a n 1 a = Θ(1) if 0 < a < 1 = n + 1 = Θ(n) if a = 1 i=0 a n+1 1 a 1 = Θ(a n ) if 1 < a (5) n i=0 ai is called geometric series. H(n) = 1 + 1/2 + 1/3 + + 1/n = n i=1 1 i = Θ(ln n) (6) H(n) is called Harmonic series. How to compute H(n)? c Jinhui Xu (University at Buffalo) CSE 431/531 Algorithm Analysis and Design 24 / 36

Integration Method Integration Method Let f (x) be an increasing function. Then for any a b: b a 1 f (x)dx b f (i) i=a b+1 a f (x)dx In the Fig, = the area of the staircase region. The first = the area of the shaded region. Since f (x) is increasing, the first holds. f(b) f(b 1) f(x) f(a) x a 1 a a+1 b 1 b c Jinhui Xu (University at Buffalo) CSE 431/531 Algorithm Analysis and Design 26 / 36

Integration Method In the Fig, is the area of the staircase region. The second is the area of the shaded region. Since f (x) is increasing, the second holds. f(b+1) f(b) f(x) f(a) a a+1 b b+1 x Similarly: Let f (x) be a decreasing function. Then for any a b: b a 1 f (x)dx b f (i) i=a b+1 a f (x)dx c Jinhui Xu (University at Buffalo) CSE 431/531 Algorithm Analysis and Design 27 / 36

Example Example For any k > 0, f (x) = x k is an increasing function. Let a = 1 and b = n. n 0 x k dx n i k i=1 n+1 1 x k dx n 0 xk dx = 1 k+1 xk+1 x=n x=0 = nk+1 k+1 ; n+1 1 x k dx = 1 k+1 xk+1 x=n+1 x=1 = (n+1)k+1 1 k+1 Thus: n k+1 k + 1 n i=1 i k (n + 1)k+1 1 k + 1 By limit test, both lower and upper bounds = Θ(n k+1 ). Thus n i=1 ik = Θ(n k+1 ). c Jinhui Xu (University at Buffalo) CSE 431/531 Algorithm Analysis and Design 28 / 36

Example f (x) = 1 x is a decreasing function. Let a = 1 and b = n. n 0 n 0 dx x n i=1 1 n+1 i dx 1 x dx x = ln x n 0 = ln n ( ) =. This doesn t work! Try a = 2 and b = n: n 1 dx x n i=2 1 n+1 i dx 2 x This gives: (ln n ln 1) n i=2 1 i (ln(n + 1) ln 2). Note ln 1 = 0. Add 1 to the above: 1 + ln n n i=1 1 i (ln(n + 1) ln 2 + 1). By limit test, both lower and upper bounds = Θ(ln n). So H(n) = n i=1 1 i = Θ(ln n) Note: lim n (ln n n i=1 1 i ) = c, where c = 0.577... is Euler constant. c Jinhui Xu (University at Buffalo) CSE 431/531 Algorithm Analysis and Design 29 / 36

Solving Linear Recursive Equations Fibonacci number Fib 0 = 0, Fib 1 = 1,..., Fib n+2 = Fib n+1 + Fib n How to compute Fib n directly from n? ( Fib n = 1 5 1 + ) n ( 5 1 2 5 1 ) n 5 2 1± 5 2 are the two roots of the equation: x 2 = x + 1. Since 1 5 < 1, the second term 0. So Fib n 1 2 (α = 1+ 5 2 = 1.618... is called the golden ratio.) For n = 8, Fib 8 = 21 where 1 5 α n = 21.0095. How to find such formula? ( 1+ 5 5 2 c Jinhui Xu (University at Buffalo) CSE 431/531 Algorithm Analysis and Design 31 / 36 ) n.

Linear recursive sequences Linear recursive sequences A sequence {f 0, f 1,..., f n...} is called a linear recursive sequence of order k if it is defined as follows: f 0, f 1,..., f k 1 are given. For all n 0, f n+k = c k 1 f n+k 1 + c k 2 f n+k 2 + + c 1 f n+1 + c 0 f n where c k 1, c k 2,... c 0 are fixed constants. Example 1: {Fib n } is a linear recursive sequence of order 2 where c 1 = 1 and c 0 = 1. Example 2: f 0 = 1, f 1 = 2, f 2 = 4 and for all n 0, f n+3 = 3f n+1 2f n Then {f n } is a linear recursive sequence of order 3 where c 2 = 0, c 1 = 3 and c 0 = 2. c Jinhui Xu (University at Buffalo) CSE 431/531 Algorithm Analysis and Design 32 / 36

Solving linear recursive sequences The characteristic equation of the linear recursive seq is: x k = c k 1 x k 1 + c k 2 x k 2 + + c 1 x 1 + c 0 Solve this equation for x. Let α 1,..., α k be the roots. Assuming all roots are distinct. Then the solution of f n has the form f n = a 1 (α 1 ) n + a 2 (α 2 ) n + + a k (α k ) n for some constants a 1, a 2,..., a k. Plug in the initial values f 0, f 1,..., f k 1, we get k equations. Solve them to find a 1, a 2,..., a k. c Jinhui Xu (University at Buffalo) CSE 431/531 Algorithm Analysis and Design 33 / 36

Example Fibonacci number F 0 = 0, F 1 = 1 and F n+2 = F n+1 + F n The characteristic equation is: x 2 = x + 1. The roots are: α 1 = 1+ 5 2 and α 2 = 1 5 2 The solution has the form: F n = a 1 (α 1 ) n + a 2 (α 2 ) n Plug in initial value F 0 = 0 and F 1 = 1: 0 = F 0 = a 1 α 0 1 + a 2α 0 2 = a 1 + a 2 1 = F 1 = a 1 α 1 1 + a 2 α 1 2 = a 1 1 + 5 2 + a 2 1 5 2 Solving this equation system, we get: a 1 = 1 5 and a 2 = 1 5. Thus: ( F n = 1 5 1 + ) n ( 5 1 2 5 1 ) n 5 2 c Jinhui Xu (University at Buffalo) CSE 431/531 Algorithm Analysis and Design 34 / 36

Solving linear recursive sequences The roots α 1,..., α k of the characteristic equ are not distinct. Say α 1 = α 2 =... = α t repeats t times. Then in the solution formula, the portion a 1 (α 1 ) n + a 2 (α 2 ) n + + a t (α t ) n is replaced by: a 1 (α 1 ) n + a 2 n 1 (α 1 ) n + + a t n t 1 (α 1 ) n Other steps are the same. c Jinhui Xu (University at Buffalo) CSE 431/531 Algorithm Analysis and Design 35 / 36

Example Example F 0 = 1, F 1 = 2, F 2 = 4 and for all n 0, F n+3 = 3F n+1 2F n The characteristic equation is: x 3 = 3x 2 or f (x) = x 3 3x + 2 = 0. To solve f (x) = 0, try x = 1. We find x = 1 is a root. So (x 1) is a factor of f (x). Thus f (x) = (x 1)(x 2 + x 2) = (x 1)(x 1)(x + 2). So the roots are α 1 = α 2 = 1 and α 3 = 2. The solution has the form: F n = a 1 1 n + a 2 n 1 n + a 3 ( 2) n. Plug in initial values: 1 = F 0 = a 1 + 0 + a 3 2 = F 1 = a 1 + a 2 2a 3 4 = F 2 = a 1 + 2a 2 + 4a 3 Solving this: a 1 = 8/9, a 2 = 4/3 and a 3 = 1/9 Thus: F n = 8 9 + 4 3 n + 1 9 ( 2)n c Jinhui Xu (University at Buffalo) CSE 431/531 Algorithm Analysis and Design 36 / 36