Fall 2014 Unit 4: Basics of complexity analysis
Correctness and efficiency So far, we have talked about correctness and termination of algorithms What about efficiency?
Running time of an algorithm For a Turing machine, the number of transitions taken until an accept or reject state is reached For a structured program, the number of evaluations of basic operations (skip, assignments, and basic arithmetic operations)
Polynomial time Brute force: For many non-trivial problems, there is a natural brute force search algorithm that checks every possible solution. Typically takes 2 N time or worse for inputs of size N.
Polynomial time There exists constants c > 0 and d > 0 such that on every input of size N, its running time is bounded by c N d steps. When the input size doubles, the algorithm only slows down by a constant factor.
Worst-case analysis Worst case running time: Obtain bound on largest possible running time of algorithm on input of a given size N Downside: Draconian view.
Worst-case analysis Worst case running time: Obtain bound on largest possible running time of algorithm on input of a given size N Downside: Draconian view. Average case running time: Obtain bound on running time of algorithm on random input as a function of input size N. Downside: Hard to model real instances by random distributions.
Worst-case polynomial time An algorithm is efficient if its running time is polynomial. In practice, poly-time algorithms almost always have low constants and low exponents Breaking through the exponential barrier of brute force typically exposes some crucial structure of the problem. Exceptions: Some poly-time algorithms useless in practice Some exponential-time algorithms widely used as worst-case instances are rare
Asymptotic order of growth Assumptions: All functions f (n) that we consider have the properties: f (n) > 1 lim n f (n) is
Asymptotic order of growth Upper bounds: T (n) is O(f (n)) if there exist constants c > 0 and n 0 0 such that for all n n 0 we have T (n) c f (n) Lower bounds: T (n) is Ω(f (n)) if there exist constants c > 0 and n 0 0 such that for all n n 0 we have T (n) c f (n). Tight bounds: T (n) is Θ(f (n)) if T (n) is both O(f (n)) and Ω(f (n)).
Examples T (n) = 32n 2 + 17n + 32 T (n) is O(n 2 ), O(n 3 ), Ω(n 2 ), Ω(n), and Θ(n 2 ) T (n) is not O(n), Ω(n 3 ), Θ(n), or Ω(n 3 )
Notation Slight abuse of notation: T (n) = O(f (n)) Asymmetric: f (n) = 5n 3 ; g(n) = 3n 2 f (n) = O(n 3 ) = g(n)... even though f (n) g(n) What I actually mean: T (n) O(f (n))
Properties: Transitivity If f = O(g) and g = O(h) then f = O(h) If f = Ω(g) and g = Ω(h) then f = Ω(h) If f = Θ(g) and g = Θ(h) then f = Θ(h)
Properties: Additivity If f = O(h) and g = O(h) then f + g = O(h) If f = Ω(h) and g = Ω(h) then f + g = Ω(h) If f = Θ(h) and g = Θ(h) then f + g = Θ(h)
Bounds through limits Let f (n) and g(n) be two positive-valued functions such that f (n) lim n g(n) exists and equals some number c > 0. Then f (n) = Θ(g(n)).
Complexity analysis: rules of thumb For all b > 1 and d > 0, we have log b n = O(n d ) For all r > 1 and d > 0, we have n d = O(r n )
Common runtimes Linear time O(n log n) time Quadratic time Cubic time O(n k ) time Exponential time Sublinear time Check out http://bigocheatsheet.com/!
Exercise: Upper and lower bounds Input: an array A with integer entries A[1],..., A[n] Output: a 2-D array B such that B[i, j] contains the sum A[i] + A[i + 1] + + A[j] 1 for i=1 to n 2 for j = i+1 to n { 3 Add up entries A[i] through A[j] 4 Store result in B[i,j] 5 } 6 } Obtain an upper bound and a lower bound for the algorithm.
Lower bound proof Consider the times during the execution of the algorithm when i n 4 and j 3n 4. In these cases, j i + 1 3n 4 n 4 + 1 > n 2. Therefore, adding up the array entries A[i] through A[j] would require at least n 2 operations, since there are more then n 2 terms to add up. There are ( n 4 )2 pairs (i, j) with i n 4 and j 3n 4. The algorithm enumerates over all of them. Therefore, the algorithm must perform at least n 2 ( n 4 )2 = n3 32 operations. This is Ω(n 3 ).
Complexity analysis of recursive algorithms Key tool: Recurrence equations
Divide-and-conquer algorithms Applicable even when a polynomial-time brute-force algorithm exists: Break up problem of size n into multiple parts of size n/k Solve the k parts recursively Combine two solutions into overall solution
Mergesort Divide array of size n into two halves Recursively sort each half Merge two halves (in n steps) to make sorted whole.
Complexity of mergesort Let T (n) = number of comparisons to mergesort an input of size n { 0 if n = 1 T (n) T ( n/2 ) + T ( n/2 ) + n otherwise
Complexity of mergesort Let T (n) = number of comparisons to mergesort an input of size n { 0 if n = 1 T (n) T ( n/2 ) + T ( n/2 ) + n otherwise Solution T (n) = O(n log 2 n) Read chapter on recurrences in Erickson s text: http://www.cs.uiuc.edu/ jeffe/teaching/algorithms/notes/ 99-recurrences.pdf
Proof by recursion tree Proof by Recursion Tree Assuming n is a power of 2: {" 0 if n =1 $ 0 if n = 1 T (n) T(n) = # 2T(n /2) + n $ T (n/2) + T (n/2) otherwise + n otherwise % sorting both halves merging T(n) n T(n/2) T(n/2) 2(n/2) T(n/4) T(n/4) T(n/4) T(n/4) log 2 n 4(n/4)... T(n / 2 k ) 2 k (n / 2 k )... T(2) T(2) T(2) T(2) T(2) T(2) T(2) T(2) n/2 (2) n log 2 n 8
Exercises Solve the following recurrences: 1 T (n) = 4T (n/2) + n
Exercises Solve the following recurrences: 1 T (n) = 4T (n/2) + n 2 T (n) = 8T (n/2) + n 2
Exercises Solve the following recurrences: 1 T (n) = 4T (n/2) + n 2 T (n) = 8T (n/2) + n 2 3 T (n) = 3T (n/2) + n
Exercises Solve the following recurrences: 1 T (n) = 4T (n/2) + n 2 T (n) = 8T (n/2) + n 2 3 T (n) = 3T (n/2) + n 4 T (n) = nt ( n) + n
Proof by induction Claim: Suppose n is a power of 2. Then if T (n) satisfies the following recurrence, then T (n) = n log 2 n. { 0 if n = 1 T (n) T (n/2) + T (n/2) + n otherwise
Proof by induction Claim: Suppose n is a power of 2. Then if T (n) satisfies the following recurrence, then T (n) = n log 2 n. { 0 if n = 1 T (n) T (n/2) + T (n/2) + n otherwise Proof: Base case: n = 1 Inductive case: Assuming T (n) = n log n, show that T (2n) = 2n log 2 2n
The general case Claim: If T (n) satisfies the following recurrence, then T (n) n lg n. { 0 if n = 1 T (n) T ( n/2 ) + T ( n/2 ) + n otherwise
The general case Claim: If T (n) satisfies the following recurrence, then T (n) n lg n. { 0 if n = 1 T (n) T ( n/2 ) + T ( n/2 ) + n otherwise Proof by complete induction: Inductive case: T (n) T (n 1 ) + T (n 2 ) + n where n 1 = n/2, n 2 = n/2 n 1 lg n 1 + n 2 lg n 2 + n n lg n 2 + n n( lg n 1 1) + n n lg n n 2 = n/2 2 lg n /2 = 2 lg n /2 lg n 2 lg n 1
Master s theorem For recurrences of the form T (n) = at (n/b) + f (n), where a 1, b > 1 If f (n) = O(n log b a c ), where c > 0, then T (n) = Θ(n log b a ) If f (n) = Θ(n log b a ), then T (n) = Θ(n log b a log n) If f (n) = Ω(n log b a+c ), where c > 0, and if af (n/b) kf (n) for some k < 1 and all sufficiently large n, then T (n) = Θ(f (n)).
Master s theorem 1 T (n) = 9T (n/3) + n 2 T (n) = T (2n/3) + 1 3 T (n) = 3T (n/4) + n lg n 4 T (n) = 2T (n/2) + n lg n
Master s theorem 1 T (n) = 9T (n/3) + n 2 T (n) = T (2n/3) + 1 3 T (n) = 3T (n/4) + n lg n 4 T (n) = 2T (n/2) + n lg n Case 3 does not apply! f (n) = n lg n is not Ω(n 1+c ) for any c.
Complexity: bits vs. words What are we counting while giving complexity bounds? Issues: Basic operations in executions of a structured program Number of moves in executions of a Turing machine Mergesort takes O(n lg n) comparisons But a comparison isn t constant-time in a Turing machine! lg n bitwise operations But neither can you assume arbitrary arithmetic operations
Complexity: bits vs. words What are we counting while giving complexity bounds? Issues: Basic operations in executions of a structured program Number of moves in executions of a Turing machine Mergesort takes O(n lg n) comparisons But a comparison isn t constant-time in a Turing machine! lg n bitwise operations But neither can you assume arbitrary arithmetic operations Conventions: Usual assumption: Unit-time operations on log n-bit words Exceptions: In certain problems like integer multiplication, use bit-level model