/4/3 Administrative Big O David Kauchak cs3 Spring 3 l Assignment : how d it go? l Assignment : out soon l CLRS code? l Videos Insertion-sort Insertion-sort Does it terminate?
/4/3 Insertion-sort Loop invariant Loop invariant: A statement about a loop that is true before the loop begins and after each iteration of the loop. Upon termination of the loop, the invariant should help you show something useful about the algorithm. Loop invariant? Is it correct? Can you prove it? Loop invariant Loop invariant: A statement about a loop that is true before the loop begins and after each iteration of the loop. At the start of each iteration of the for loop of lines -7 the subarray A[..j ] is the sorted version of the original elements of A[..j ] Loop invariant At the start of each iteration of the for loop of lines -7 the subarray A[..j ] is the sorted version of the original elements of A[..j ] Proof by induction - Base case: invariant is true before loop - Inductive case: it is true after each iteration Proof?
/4/3 Insertion-sort How long will it take to run? Asymptotic notation l How do you answer the question: what is the running time of algorithm x? l We need a way to talk about the computational cost of an algorithm that focuses on the essential parts and ignores irrelevant details l You ve seen some of this already: l l linear n log n l n Asymptotic notation Precisely calculating the actual steps is tedious and not generally useful Different operations take different amounts of time. Even from run to run, things such as caching, etc. cause variations We want to identify categories of algorithmic runtimes For example f n takes n steps f n takes n + steps f 3 n takes 3n+ steps Which algorithm is better? Is the difference between f and f 3 important/ significant? 3
/4/3 Runtime examples Big O: Upper bound Ogn is the set of functions: # Ogn f n : such that ' & f n cgn for all n n Big O: Upper bound Ogn is the set of functions: # Ogn f n : such that ' & f n cgn for all n n Big O: Upper bound Ogn is the set of functions: # Ogn f n : such that ' & f n cgn for all n n We can bound the function fn above by some constant factor of gn We can bound the function fn above by some constant multiplied by gn For some increasing range 4
/4/3 Big O: Upper bound Ogn is the set of functions: # Ogn f n : such that ' & f n cgn for all n n Big O: Upper bound Ogn is the set of functions: # Ogn f n : such that ' & f n cgn for all n n O n 3 4 3n / n + n + 5n + 4 6n Generally, we re most interested in big O notation since it is an upper bound on the running time Omega: Lower bound Ωgn is the set of functions: & Ωgn f n : such that & '& cgn f n for all n n Omega: Lower bound Ωgn is the set of functions: & Ωgn f n : such that & '& cgn f n for all n n We can bound the function fn below by some constant factor of gn 5
/4/3 Omega: Lower bound Ωgn is the set of functions: & Ωgn f n : such that & '& cgn f n for all n n Theta: Upper and lower bound Θgn is the set of functions: & there exists positive constants c Θgn f n :, c and n such that & '& c gn f n c gn for all n n Ω n 3 4 3n / n 6n + n + 5n + 4 3 Theta: Upper and lower bound Θgn is the set of functions: & there exists positive constants c Θgn f n :,c and n such that & '& c gn f n c gn for all n n Theta: Upper and lower bound Θgn is the set of functions: & there exists positive constants c Θgn f n :, c and n such that & '& c gn f n c gn for all n n We can bound the function fn above and below by some constant factor of gn though different constants Note: A function is theta bounded iff it is big O bounded and Omega bounded 6
/4/3 Theta: Upper and lower bound Visually Θgn is the set of functions: & there exists positive constants c Θgn f n :,c and n such that & '& c gn f n c gn for all n n Θ n 3 4 3n / n + n + 5n + 4 3n + nlog n fn Visually: upper bound Visually: lower bound fn fn n n 7
/4/3 worst-case vs. best-case vs. average-case worst-case: what is the worst the running time of the algorithm can be? best-case: what is the best the running time of the algorithm can be? average-case: given random data, what is the running time of the algorithm? Don t confuse this with O, Ω and Θ. The cases above are situations, asymptotic notation is about bounding particular situations Proving bounds: find constants that satisfy inequalities Show that 5n 5n + is Θn Step : Prove On Find constants c and n such that 5n 5n + cn for all n > n cn 5n 5n + c 5 5 / n + / n Let n and c 5 + 5. /n only get smaller as n increases and we ignore -5/n since it only varies between -5 and Proving bounds Step : Prove Ωn Find constants c and n such that 5n 5n + cn for all n > n cn 5n 5n + c 5 5 / n + / n Bounds Is 5n On? No How would we prove it? # Ogn f n : such that ' & f n cgn for all n n Let n 4 and c 5 5/4.5 or anything less than.5. 5/n is always decreasing and we ignore /n since it is always between and. 8
/4/3 Disproving bounds Some rules of thumb Is 5n On? # Ogn f n : & Assume it s true. such that ' f n cgn for all n n Multiplicative constants can be omitted l 4n becomes n l 7 log n become log n Lower order functions can be omitted l n + 5 becomes n l n + n becomes n That means there exists some c and n such that 5n cn for n > n 5n c contradiction! n a dominates n b if a > b l n dominates n, so n +n becomes n l n.5 dominates n.4 Some rules of thumb a n dominates b n if a > b l 3 n dominates n Any exponential dominates any polynomial l 3 n dominates n 5 l n dominates n c Any polynomial dominates any logorithm l n dominates log n or log log n l n dominates n log n l n / dominates log n Do not omit lower order terms of different variables n + m does not become n Big O n + n log n + 5 n -5n + n 3 log n n log n + n + 5n 3 n 5 + n! + n n 9
/4/3 Some examples l O constant. Fixed amount of work, regardless of the input size l add two 3 bit numbers l determine if a number is even or odd l sum the first elements of an array l delete an element from a doubly linked list l Olog n logarithmic. At each iteration, discards some portion of the input i.e. half l binary search Some examples l On linear. Do a constant amount of work on each element of the input l find an item in a linked list l determine the largest element in an array l On log n log-linear. Divide and conquer algorithms with a linear amount of work to recombine l Sort a list of number with MergeSort l FFT Some examples l On quadratic. Double nested loops that iterate over the data l Insertion sort l O n exponential l Enumerate all possible subsets l Traveling salesman using dynamic programming l On! l Enumerate all permutations l determinant of a matrix with expansion by minors