Analysis of Algorithms - Using Asymptotic Bounds - Andreas Ermedahl MRTC (Mälardalens Real-Time Research Center) andreas.ermedahl@mdh.se Autumn 004 Rehersal: Asymptotic bounds Gives running time bounds for algorithms Examples: f(n) = O(n ) means that: n is an asymptotic upper bound for f(n) f(n) = Ω(n) means that: n is an asymptotic lower bound for f(n) f(n) is the real execution time of the analyzed algorithm Bounds valid for large sizes of input n f(n) = Θ(n log n) means that: n log n is a lower and upper bound for f(n) We can use asymptotic bounds for individual code parts to derive asymptotic bounds for the overall algorithm Θ(g(n)) = { f(n) : there exist positive constants c 1, c, and n 0 such that 0 c 1 g(n) f(n) c g(n) for all n n 0 } Example: Single Loop Algorithm to sum all elements in an array A[ ] COST TIMES BOUNDS 1. for j 1 to length[a] c 1 n+1 Θ(n). do sum sum + A[j] c n Θ(n). return sum c 1 Θ(1) Total execution time T(n): T(n) = (c 1 n + c 1 ) + (c n) + (c ) = = Θ(n) + Θ(n) + Θ(1) = Θ(n) fl Running time is Θ(n) We can derive a bound for each stmt in isolation The individual stmt bounds together give an overall bound Asymptotic bounds & inputs Θ-notation useful for algorithms whose running time does not vary with indata for the same input size For example, MERGESORT has: T(n) = Θ(n log n) O- and Ω-notation useful for algorithms where running time varies with indata for same input size For example, INSERTIONSORT has: T(n) = Ω(n) and T(n) = O(n ) There exists no function g(n) such that T(n) = Θ(g(n)) Example: best- & worst- case Only add array elements fulfilling special property COST TIMES BOUNDS 1. for j 1 to length[a]-1 c 1 n Θ(n). if A[j] A[j-1] c n-1 Θ(n). then sum sum + A[j] c 4. return sum c 4 1 Θ(1) Best case time is when c is never executed: 0 T best (n) = Θ(n) + Θ(n) + 0 + Θ(1) = Θ(n) fl T(n) = Ω(n) Worst case time is when c is always executed: n-1 T worst (n) = Θ(n) + Θ(n) + Θ(n) + Θ(1) = Θ(n) fl T(n) = O(n) fl Since T(n) = Ω(n) and T(n) = O(n) then T(n) = Θ(n) Example: MERGESORT MERGESORT(A, p, r) BOUNDS 1. if p < r Θ(1). then q (p + r)/ Θ(1). MERGESORT(A, p, q) 4. MERGESORT(A, q + 1, r) 5. MERGE(A, p, q, r) Θ(n) MERGE function code consists of several statements When called MERGE runs in linear time Gives a Θ(n) cost for MERGE in calculation for MERGESORT Constant time for base case: Θ(1) Constant time for division: Θ(1)
Analysis of Algorithms - Recurrences - Andreas Ermedahl MRTC (Mälardalens Real-Time Research Center) andreas.ermedahl@mdh.se Autumn 004 Recurrences To be able to compare and evaluate algorithms we need a way to describe their execution times We often like to describe the execution time of an algorithm as a function T(n) of its input size n When an algorithm contains a recursive call to itself its running time T(n) can often be described by a recurrence A recurrence is an equation or inequality that describes a function in terms of: one or more base cases, and the same function called with smaller argument Example: MERGESORT The derived recurrence for the MERGESORT running time T(n): Θ(1) ( n / ) + Θ(1) + Θ( n) a = number of subproblems = n/ = size per subproblem MERGESORT(A, p, r) 1. if p < r Base case. then q (p + r)/ Divide. MERGESORT(A, p, q) Conquer 4. MERGESORT(A, q + 1, r) Conquer 5. MERGE(A, p, q, r) Combine Constant cost for dividing subproblems Base: sort only one element otherwise Time for merging resulting subproblems Solution: T(n) = Θ(n lg n) More Examples: ( n 1) + 1 Solution: T(n) = n ( n / ) + n Solution: T(n) = n lg n + n More Examples (cont): 0 if n = ( n) + 1 if n > Solution: T(n) = n lg lg n ( n / ) + T (n / ) + n Solution: T(n) = Θ(n lg n) Solving recurrences Interesting to solve (or bound) recurrences We will demonstrate some methods for this: The Substitution method - guess a solution and use mathematical induction to show that it works The Master method provides bounds for recurrences of the form: T(n) = at(n/b) + f(n) where a 1, b > 1 and f(n) is a given function The Recursion-tree method convert the recurrence into a tree whose nodes represent costs incurred at various levels of the recursion Tree method used to generate a good guess! Note! Tree method does not generate a mathematical proof! Solution must still be proven, eg. using substitution method
Technicalities: Floors and ceilings We usually assume that all arguments are integers We often write T(f(n)) when we mean T( f(n) ) or T( f(n) ) Floors and ceilings are usually ignored since they only have a constant effect! For example, recurrence for MERGESORT is really: Θ(1) ( n / ) + T ( n / ) + Θ(1) + Θ( n) It is not certain that n is even! Eg. if n=11 then we get subparts of sizes 5 and 6 For all real x holds that: x-1 < x x x < x+1 We use T(n/) since constant effect only! Technicalities: Boundary conditions We usually express both the recurrence and its solution using asymptotic notation We then typically omit boundary conditions for the recurrence, Assumes solution bound by constant for sufficiently small argument For example, we normally state MERGESORT recurrence as: = T ( n / ) + Θ( n) Base-case T(1) omitted! When we desire an exact, rather than an asymptotic, solution, we need to deal with boundary conditions In practice, we just use asymptotic notations most of the time, and ignore the boundary conditions The substitution method To mathematically prove a guessed solution Requires often some effort and careful analysis to guarantee correctness Works in two steps: 1. Guess the form of the solution (bound). Use mathematical induction to find constants and show that the solution (bound) works The substitution method Example: find solution to T(n) recurrence given by: ( n / ) + n Note! No asymptotics in recurrence! 1. Guess: T(n) = n lg n + n Exact solution wanted!. Induction: Base: n =1 fl n lg n + n = 0+1 = 1 = T(n) ok! Inductive step: Inductive hypothesis is that T(k) = k lg k + k for all k < n. - We ll use this inductive hypothesis for T(n/) ( n / ) + n The substitution method Inductive hypothesis: T(k) = k lg k + k for all k < n We ll use this inductive hypothesis for T(n/) T(n) = T(n/) + n Using the induction hypothesis! = ((n/) lg (n/) + (n/)) + n = n lg n/ + n + n lg a/b = lg a lg b = n(lg n - lg ) + n + n = n lg n - n + n + n lg = 1 = n lg n + n The guess was correct! The substitution method The example did not use asymptotic notations Neither in the recurrence or in the solution Generally, we use asymptotic notations: We write the recurrence as: T(n) = T(n/) + Θ(n) We express the solution as: T(n) = O(n lg n) We could use the substitution method to verify guessed asymptotic bounds
The substitution method Example: find asymptotic upper bound to T(n) given by: ( n / ) + n Guess: T(n) = O(n lg n) Method: Prove by induction that there exists a constant c > 0 and some constant n 0 such that T(n) lg n for all n n 0 By definition of O The substitution method (cont) Mission: Find a n 0 and corresponding c so that T(n) lg n formula holds Base: (first try) set n 0 =1 Recurrence formula gives: T(1)=1 T(n) lg n formula gives T(1) c1 lg 1 c1 * 0 = 0 fl We have to find another base (ie another value for n 0 )! Base: (second try) set n 0 = Recurrence formula gives: T()=T( / )+=T(1)+=4 T(n) lg n formula gives T() c lg = c fl ( n / ) + n Any choice of c will suffice to make the choice of n 0 = to hold Conclusion: c = and n 0 = is a base for the induction n 0 should be found! The substitution method (cont) Induction assumption: T(k) ck lg k bound holds for k = n/ I.e. T( n/ ) c n/ lg( n/ ) is assumed to hold We want to show that it holds for T(n) Using the induction assumption Proof: Substituting into the recurrence yields: ( n / ) + n T(n) = T( n/ ) + n [using induction assumption] (c n/ lg ( n/ )) + n [since n/ n and lg( n/ ) lg(n/)] lg (n/) + n [since lg(n/) = lg n lg ] = lg n lg + n [since lg = 1] = lg n + n [since n if c 1] lg n Conclusion: T(n) = O(n lg n) since it holds that T(n) lg n for c= and all n n 0 = Expanding the recurrence An obvious idea: Expand (iterate) recurrence to a summation formula dependent only on n and initial conditions Use techniques for bounding sums to bound the solution for the recurrence Example: T(n) = T(n/4)+ Θ(n ) Generates three subproblems of 1/4 the size of the original subproblem Cost to divide problem into subproblem and merge results of subproblems bounded by Θ(n ) Expanding the recurrence (cont.) We rewrite the recurrence as follows: T(n) = T(n/4) + Expansion of T(n) gives: T(n) = + T(n/4) = = + ( + T(n/4/4)) = = + + 9T(n/16) = = + + 9c(n/16) + 7T(n/64) = = We can see a pattern in the terms generated in the summation, but it is hard to directly derive a bound We make use of a recursion tree to vizualize and increase the understanding Recursion trees Tree visualizing the recursive calls in the recurrence Useful for deriving a good guess for substitution method Has one node for each time a recursive call is made (expanding multiples of calls) Each node represent the cost of a single subproblem Typical steps performed: 1. estimate number of levels in the tree /. sum horizontally for each level in the tree. finally sum vertically to derive total cost Unbalanced trees still give upper bounds, if maximum depth is used as limit for vertical summation / /4 /4 /4 /4 c c c c c... c c
Cost to sort n elements T(n) Size of subproblem at depth i is n/4 i Recursion tree: Expanding expand Cost to sort n/4 elements T(n/4) Three subproblems each of size n/4 T(n/4) T(n/4) expand Cost to divide n into subproblems and combine their results T(n)=T(n/4)+ Cost to divide n/4 into subproblems and combine their results expand levels Recurrence tree: Final tree Question: 1. How many levels has the tree c(n/16) c(n/16) c(n/16) c(n/16) c(n/16) c(n/16) c(n/16) c(n/16) c(n/16) T(n)=T(n/4)+ cost expand T(n/16) T(n/16) T(n/16) T(n/16) T(n/16) T(n/16) T(n/16) T(n/16) T(n/16) T(1) T(1) T(1) T(1) T(1) T(1)T(1) T(1) T(1) T(1) T(1)T(1)... T(1) T(1)T(1) T(1) 9 Nine subproblems each of size n/16 total: 1. Estimate height of tree Size of subproblem decrease with the distance to the root The leaf nodes appear when subproblem size is 1 At each level the size of the problem is divided by 4 Size of subproblem at depth i is n/4 i Thus, subproblem size becomes 1 when n/4 i = 1 ñ n = 4 i This is equivalent to when i = log 4 n The final tree therefore has height log 4 n and log 4 n + 1 levels Recurrence tree: Final tree Question:. What is the cost of the different tree levels levels c(n/16) c(n/16) c(n/16) c(n/16) c(n/16) c(n/16) c(n/16) c(n/16) c(n/16) log 4 n +1 T(n)=T(n/4)+ T(1) T(1) T(1) T(1) T(1) T(1)T(1) T(1) T(1) T(1) T(1)T(1)... T(1) T(1)T(1) T(1) cost total:. Cost for levels in tree We start by investigating all nodes except the leaves Each level has three times more nodes than the level above The number of nodes at level i are therefore i At each level the size of subproblem is reduced by a factor of 4 A node at level i has a cost of c(n/4 i ) The total cost for a level = sum of cost of all nodes at that level Thus, the total cost for the nodes at level i is therefore: i * c(n/4i ) = (/16) i Recurrence tree: Final tree Question:. What is the cost for the leafs levels c(n/16) c(n/16) c(n/16) c(n/16) c(n/16) c(n/16) c(n/16) c(n/16) c(n/16) log 4 n +1 T(n)=T(n/4)+ T(1) T(1) T(1) T(1) T(1) T(1)T(1) T(1) T(1) T(1) T(1)T(1)... T(1) T(1)T(1) T(1) cost total: (/16) (/16)
. Cost for leaf nodes We also need to decide the cost for the leaf nodes The last level (contaning all the leafs) has depth log 4 n The total number of nodes at the last level are therefore: log 4n = n log 4 (Using fact: log 4n = 4 log 4 log4n = 4 log 4*log 4n = 4 log 4n log4 =n log 4 ) Each node contributes with cost: T(1) The total cost for all the nodes at the last level of the tree is therefore: n log4 T(1) = Θ(n log4 ) Assuming T(1) is bound by some constant Recurrence tree: Example (cont.) Question:. What is the total cost for the whole tree levels c(n/16) c(n/16) c(n/16) c(n/16) c(n/16) c(n/16) c(n/16) c(n/16) c(n/16) log 4 n +1 n log4 T(n)=T(n/4)+ T(1) T(1) T(1) T(1) T(1) T(1)T(1) T(1) T(1) T(1) T(1)T(1)... T(1) T(1)T(1) T(1) cost total: (/16) (/16) Θ(n log4 ). Total cost for whole tree We sum over all costs over all levels in the tree We obtain: Cost for leaves log4 n 1 log4 T(n) = + + + + + Θ( 16 16 16 log4 n 1 i log4 Cost for nodes = + Θ( í 0 16 above the leaves = i log4 < + Θ( Geometric series: (A.6) í = 0 16 k 1 1 x = log4 = + Θ( k = 0 1 x 1 (/16) 16 log4 = 1 + Θ( Resulting guess for: = O( T(n) = T(n/4) + Proving the obtained bound We derived a O(n ) guess for T(n)=T(n/4)+ We have not proved the bound yet! From the definition Proof by substitution method: of O-notation We want to show that T(n) dn for some constant d > 0 T( n) T ( n/ 4) + d( n/ 4) = dn + 16 dn Last step holds if d (16/1)c + Induction assumption: T(k) dk holds for k < n Set k = n/4 and use induction assumption Unbalanced trees Some recurrences gives unbalanced recursion trees Eg. recurrence T(n) = T(n/)+T(n/)+ gives tree: c(n/9) c(n/) Leftmost branch peters out after log n levels c(n/9) c(n/9) c(n/) c(4n/9) Each full level has cost Each non-full level has cost Rightmost branch peters out after log / n levels T(n) = T(n/)+T(n/)+ Unbalanced tree (cont.) The tree has log n full levels After log / n levels, the problem size is down to 1 Each level contributes Lower bound guess: T(n) dn log n = Ω(n lg n) c(n/) c(n/9) c(n/) c(n/9) c(n/9) c(4n/9) Upper bound guess: Min levels T(n) dn log / n = O(n lg n) Max levels Guesses could be proven by substitution method See course book for details
The Master method Cookbook method to solve recurrences on form: T(n) = at(n/b) + f(n) Suitable for n > 0, a 1 and b>1 For solving the kind of recurrences which often appear for divide-and-conquer algorithms a is the number of subproblems created n/b is the size of each subproblem f(n) is a function for the cost for dividing problem into subproblems and combining their solutions Can be applied in three different cases Depends on how n log b a compares with f(n) The largest one will dominate the solution The Master method The Master Method gives tight bounds for T(n) = at(n/b) + f(n) in three cases: Case 1: f(n) = O(n log b a - ε ) for some constant ε > 0 Gives solution: T(n) = Θ(n log b a ) Case : f(n) = Θ(n log b a ) Gives solution: T(n) = Θ(n log b a lg n) Case : f(n) = Ω(n log b a+ ε ) for some constant ε > 0 and f(n) satisfies that a f(n/b) cf(n) for some c < 1 Gives solution: T(n) = Θ(f(n)) The Master method Cases depend on how f(n) compares with n log b a Roughly speaking: Case 1 holds when: f(n) < n log b a Case holds when: f(n) = n log b a Case holds when: f(n) > n log b a But only roughly speaking! The cases do not cover all possible f(n) values Case 1: f(n) must be polynomially smaller than n log b a Case : f(n) must be polynomially larger than n log b a and fulfill extra regularity condition There are gaps between case 1 and and between case and not covered by theorem Means that: f(n) must be smaller than n log b a by a factor of n k for constant k > 0 Error in your printed slides! Master method: examples Examples of recurrences solvable by the Master method: T(n) = T(n/) + 1 T(n) = T(n/) + n T(n) = T(n/) + n Example of recurrence not solvable by the Master method: T(n) = T(n/) + log n Use the other methods instead! Master Method: Example The recurrence for MERGESORT is: T(n)=T(n/)+Θ(n) We set values as: a=, b= and f(n)=θ(n) We have that n log b a = n log = n Thus, f(n)=θ(n log b a ) and case applies We directly obtain T(n) = Θ(n log b a lg n) = Θ(n lg n) Much easier!!! More Examples T(n) = 5T(n/) + Θ(n ) log 5 - ε = for some constant ε > 0 We can use Case 1 fl T(n) = Θ(n log 5 ) T(n) = 5T(n/) + Θ(n ) log 5 - ε = for some constant > 0 Check regularity condition: a f(n/b) = 5(n/) = 5n /8 for c = 5/8 < 1 We can use Case fl T(n) = Θ(n )
Master Method The proof for the Master Method is given in section 4.4 Uses recursion tree and induction techniques You do not need to understand the proof to apply the method Just skim through section 4.4 Summary Recurrences appear frequently in running time formulas for recursive algorithms Three methods presented for solving such recurrences: The Substitution method The Recursion-tree method The Master method Next lecture: HEAPSORT The End!