Lecture 2. Fundamentals of the Analysis of Algorithm Efficiency

Lecture 2 Fundamentals of the Analysis of Algorithm Efficiency 1

Lecture Contents 1. Analysis Framework 2. Asymptotic Notations and Basic Efficiency Classes 3. Mathematical Analysis of Nonrecursive Algorithms 4. Mathematical Analysis of Recursive Algorithms 5. Example: Fibonacci Numbers 6. Empirical Analysis of Algorithms 7. Algorithm Visualization 2

Analysis of Algorithms Issues: - correctness - time efficiency - space efficiency - optimality Approaches: - theoretical analysis - empirical analysis 3

Theoretical Analysis of Time Efficiency Time efficiency is analyzed by determining the number of repetitions of the basic operation as a function T(n) of input size n Basic operation: the operation that contributes most towards the running time of the algorithm 4

Theoretical Analysis of Time Efficiency input size T(n) c op C(n) running time execution time for basic operation Number of times basic operation is executed 5

Input Size and Basic Operation Examples Problem Input size measure Basic operation Searching for key in a list of n items Number of list s items, i.e. n Key comparison Multiplication of two matrices Matrix dimensions or total number of elements Multiplication of two numbers Checking primality of a given integer n n s size = number of digits (in binary representation) Division Typical graph problem #vertices and/or edges Visiting a vertex or traversing an edge 6

Empirical Analysis of Time Efficiency Select a specific (typical) sample of inputs Use physical unit of time (e.g., milliseconds) or Count actual number of basic operation s executions Analyze the empirical data 7

Worst-case, Best-case, Average-case Efficiencies For some algorithms, efficiency depends on form of input, so we need to do analysis for - Worst case C worst (n): determine the kind of inputs for which the count C(n) will be the largest among all possible inputs of size n. - Best case C best (n): determine the kind of inputs for which the count C(n) will be the smallest among all possible inputs of size n. 8

Worst-case, Best-case, Average-case Efficiencies - Average case: C avg (n) NOT the average of worst and best cases Expected number of basic operations considered as a random variable under some assumption about the probability distribution of all possible inputs 9

Example: Sequential Search Algorithm SequentialSearch(A[0..n 1], K) //Searches for given value in given array by sequential search //Input: Array A[0..n 1] and search key K //Output: Index of 1 st element of A that matches K // or 1 if there are no matching elements i 0 while i < n and A[i] K do i i + 1 if i < n return i else return 1 10

Example: Sequential Search Worst case C worst (n) = n The last element is equal to a search key or unsuccessful Best case C best (n) = 1 The first element is equal to a search key 11

Example: Sequential Search Average case - We have the following assumptions: a. probability of successful search is p (0 p 1) b. probability of first match occurring in ith position is the same for every i. 12

Example: Sequential Search Average case - successful search, probability of first match occurring in ith position is p/n for every i, and #comparisons made by the algorithm in such a situation is obviously i. - unsuccessful search, #comparisons is n with probability of such a search being (1 p). 13

Example: Sequential Search Average case p = 1 (successful), C avg (n) = (n + 1)/2 p = 0 (unsuccessful), C avg (n) = n 14

Types of Formulas for Basic Operation s Count Exact formula e.g., C(n) = n(n-1)/2 Approximate formula with specific multiplicative constant. e.g., C(n) 0.5 n 2 Approximate formula with unknown multiplicative constant. e.g., C(n) cn 2 15

Order of Growth Efficiency analysis ignores a multiplicative constant and concentrates on the count s order of growth for large input sizes (i.e., n ) i.e., log 2 n < n < n log 2 n < n 2 < n 3 < 2 n < n! for n large enough 16

Values of Some Important Functions as n 17

Asymptotic Order of Growth A way of comparing functions that ignores constant factors and small input sizes O(g(n)): set of functions t(n) that grow no faster than g(n) Ω(g(n)): set of functions t(n) that grow at least as fast as g(n) Θ(g(n)): set of functions t(n) that grow at same rate as g(n) 19

Formal Definition of Big-Oh O A function t(n) is said to be in O(g(n)), denoted as t(n) O(g(n)), if t(n) is bounded above by some constant multiple of g(n) for all large n, i.e., if there exist some positive constant c and some nonnegative integer n 0 such that t(n) cg(n) for all n n 0. // g(n) is upper bound of t(n) Examples: n O(n 2 ) and 100n + 5 O(n 2 ), n(n - 1)/2 O(n 2 ) 20

Big-Oh. Figure 2.1 Big-oh notation: t(n) O(g(n)). 21

Formal Definition of Big-Omega Ω A function t(n) is said to be in Ω(g(n)), denoted as t(n) Ω(g(n)), if t(n) is bounded below by some positive constant multiple of g(n) for all large n, i.e., if there exist some positive constant c and some nonnegative integer n 0 such that t(n) cg(n) for all n n 0. // g(n) is lower bound of t(n) Examples: n 3 Ω(n 2 ), n(n 1)/2 Ω(n 2 ) 22

Big-Omega. Figure 2.2 Big-omega notation: t(n) Ω(g(n)). 23

Formal Definition of Big-Theta Θ A function t(n) is said to be in (g(n)), denoted as t(n) (g(n)), if t(n) is bounded both above and below by some positive constant multiples of g(n) for all large n, i.e., if there exist some positive constant c 1 and c 2 and some nonnegative integer n 0 such that c 2 g(n) t(n) c 1 g(n) for all n n 0. Examples: n(n 1)/2 (n 2 ) 24

Big-Theta. Figure 2.3 Big-omega notation: t(n) (g(n)). 25

Some Properties of Asymptotic Order of Growth f(n) O(f(n)) f(n) O(g(n)) iff g(n) (f(n)) If f (n) O(g (n)) and g(n) O(h(n)), then f(n) O(h(n)) // Note similarity with a b If t 1 (n) O(g 1 (n)) and t 2 (n) O(g 2 (n)), then t 1 (n) + t 2 (n) O(max{g 1 (n), g 2 (n)}) (The analogous assertions are true for the Ω and notations as well.) 26

Using Limits for Comparing Order of Growth 27

L Hôpital s Rule and Stirling s Formula L Hôpital s rule // f (a): the derivative of f at a Stirling s formula 28

Example Compare the orders of growth of n(n - 1)/2 and n 2. Since the limit is equal to a positive constant, the functions have the same order of growth, or symbolically, n(n 1)/2 (n 2 ). 29

Orders of Growth of Some Important Functions All logarithmic functions log a n belong to the same class (log n) no matter what the logarithm s base a > 1 is All polynomials of the same degree k belong to the same class: a k n k + a k-1 n k-1 + + a 0 (n k ) Exponential functions a n have different orders of growth for different a s order log n < order n ( > 0) < order a n n! < order n n < order 30

Basic Asymptotic Efficiency Classes 1 constant log n logarithmic n linear n log n n-log-n n 2 n 3 quadratic cubic 2 n exponential n! factorial 31

Time Efficiency of Nonrecursive Algorithms General Plan for Analysis - Decide on parameter n indicating input size - Identify algorithm s basic operation - Determine worst, average, and best cases for input of size n - Set up a sum for the number of times the basic operation is executed - Simplify the sum using standard formulas and rules (see Appendix A) 33

Useful Summation Formulas and Rules 1 i n i 2 = 1 2 +2 2 + +n 2 = n(n+1)(2n+1)/6 n 3 /3 (n 3 ) 0 i n a i = 1 + a + + a n = (a n+1 1)/(a 1) for any a 1 0 i n 2 i = 2 0 + 2 1 + + 2 n = 2 n+1 1 (2 n ) (a i ± b i ) = a i ± b i, ca i = c a i, l i u a i = l i m a i + m+1 i u a i 34

Example 1: Maximum Element Algorithm MaxElement(A[0..n 1]) //Determines value of the largest element in a given array //Input: An array A[0..n 1] of real numbers //Output: The value of the largest element in A maxval A[0] for i 1 to n 1 do if A[i] > maxval maxval A[i] return maxval 35

Example 2: Element Uniqueness Problem Algorithm UniqueElements(A[0..n 1]) //Determines whether all elements in array are distinct //Input: An array A[0..n 1] //Output: Returns true if all elements in A are distinct and false otherwise for i 0 to n 2 do for j i + 1 to n 1 do if A[i] = A[j] return false return true 36

Example 3: Matrix Multiplication Algorithm MatrixMultiplication(A[0..n 1, 0..n 1], B[0..n 1, 0..n 1]) //Multiplies two n-by-n matrices by the definition-based algorithm //Input: Two n-by-n matrices A and B //Output: Matrix C = AB for i 0 to n 1 do for j 0 to n 1 do C[i, j] 0.0 for k 0 to n 1 do C[i, j] C[i, j] + A[i, k] B[k, j] return C 37

Example 4: Counting Binary Digits Algorithm Binary(n) //Input: A positive decimal integer n //Output: The number of binary digits in n s binary representation count 1 while n > 1 do count count + 1 n n/2 return count Prove log 2 n + 1 = log 2 (n + 1), where integer n > 0. 38

Plan for Analysis of Recursive Algorithms Decide on a parameter indicating an input s size. Identify the algorithm s basic operation. Check whether the number of times the basic operation is executed may vary on different inputs of the same size. (If it may, the worst, average, and best cases must be investigated separately.) 40

Plan for Analysis of Recursive Algorithms Set up a recurrence relation with an appropriate initial condition expressing the number of times the basic operation is executed. Solve the recurrence (or, at the very least, establish its solution s order of growth) by backward substitutions or another method. 41

Example 1: Recursive Evaluation of n! Algorithm F(n) //Computes n! recursively //Input: A nonnegative integer n //Output: The value of n! if n = 0 return 1 else return F(n 1) n 42

Solving the Recurrence for M(n) Method of backward substitutions M(n) = M(n 1) + 1 Substitute M(n 1) = M(n 2) + 1 M(n) = [M(n 2) + 1] + 1 = M(n 2) + 2 Substitute M(n 2) = M(n 3) + 1 M(n) = [M(n 3) + 1] + 2 = M(n 3) + 3... M(n) = M(n i) + i... M(n) = M(n n) + n = n (n) (initial condition M(0) = 0 for i = n) 43

Example 2: The Tower of Hanoi Puzzle 44

Solving Recurrence for Number of Moves Recurrence equation M(n) = 2M(n 1) + 1 for n > 1, M(1) = 1. // initial condition for i = n 1) Method of backward substitutions M(n) = 2M(n 1) + 1 Substitute M(n 1) = 2M(n 2) + 1 M(n) = 2[2M(n 2) + 1] + 1 = 2 2 M(n 2) + 2 + 1 Substitute M(n 2) = 2M(n 3) + 1 45

Solving Recurrence for Number of Moves M(n) = 2 2 [2M(n 3) + 1] + 2 + 1 = 2 3 M(n 3) + 2 2 + 2 + 1.... = 2 i M(n i) + 2 i-1 + 2 i-2 + + 2 + 1 = 2 i M(n i) + 2 i 1. // Appendix A... // M(1) = 1 when i = n 1 M(n) = 2 n-1 M(n (n 1)) + 2 n-1 1 M(n) = 2 n-1 M(1) + 2 n-1 1 = 2 n-1 + 2 n-1 1 = M(n) = 2 n 1 (2 n ). // exponential algorithm 46

Example 3: Count #bits Recursively Algorithm BinRec(n) //Input: A positive decimal integer n //Output: The number of binary digits in n s binary representation if n = 1 return 1 else return BinRec( n/2 ) + 1 47

Example 3: Count #bits Recursively Recurrence relation A(n) = A( n/2 ) + 1 for n > 1. A(1) = 0, // initial condition for n = 1 Let us solve the recurrence for n = 2 k A(2 k ) = A(2 k-1 ) + 1 for k > 0, A(2 0 ) = 0. // k = 0 48

Example 3: Count #bits Recursively Method of backward substitutions A(2 k ) = A(2 k-1 ) + 1 Substitute A(2 k-1 ) = A(2 k-2 ) + 1 A(2 k ) = [A(2 k-2 ) + 1] + 1 = A(2 k-2 ) + 2 Substitute A(2 k-2 ) = A(2 k-3 ) + 1 A(2 k ) = [A(2 k-3 ) + 1] + 2 = A(2 k-3 ) + 3... A(2 k ) = A(2 k-i ) + i A(2 k ) = A(2 k-k ) + k. 49

Example 3: Count #bits Recursively A(2 k ) = A(1) + k = k n = 2 k k = log 2 n A(n) = log 2 n (log n). The exact solution for an arbitrary value of n is A(n) = log 2 n. 50

Fibonacci Numbers The Fibonacci numbers: 0, 1, 1, 2, 3, 5, 8, 13, 21, Mathematical definition of Fibonacci sequence F(n) = F(n - 1) + F(n - 2) F(0) = 0, F(1) = 1 // initial conditions How to compute the nth Fibonacci number? 52

Computing the nth Fibonacci Number 1. Explicit formula ( n ), 2. Recursive definition-based algorithm ( n ) 3. Nonrecursive definition-based algorithm (n) 4. Logarithmic algorithm based on formula: F(n-1) F(n) F(n) F(n+1) = 0 1 1 1 n for n 1 (log n) 53

Explicit Formula for the nth Fibonacci Number F(n) = F(n - 1) + F(n - 2) is rewritten as F(n) - F(n - 1) - F(n - 2) = 0 Its characteristic equation is r 2 - r - 1 = 0, The roots of r 2 - r - 1 = 0 are 54

Explicit Formula for the nth Fibonacci Number According to Case 1 of Theorem 1 in Appendix B, we have With the initial conditions F(0) = 0, F(1) = 1, we have 55

Explicit Formula for the nth Fibonacci Number Substituting = - into the second equation, we get = 1/ and = -1/. 5 5 56

Explicit Formula for the nth Fibonacci Number 57

Compute the nth Fibonacci Number Recursively Algorithm F(n) //Computes the nth Fibonacci number recursively by using its definition //Input: A nonnegative integer n //Output: The nth Fibonacci number if n 1 return n else return F(n 1) + F(n 2) 58

Compute the nth Fibonacci Number Recursively The number of additions is A(n) = A(n 1) + A(n 2) + 1 for n > 1, A(0) = 0, A(l) = 0. // initial conditions Recast A(n) as follows [A(n) + 1] [A(n 1) + 1] [A(n 2) + 1] = 0 Substitue A(n) + 1 = B(n), we get B(n) B(n 1) B(n 2) = 0 B(0) = 1, B(1) = 1. 59

Compute the nth Fibonacci Number Recursively The recurrence B(n) is the same as F(n) except that B(n) starts with two ones and thus B(n) runs one step ahead of F(n). Thus, B(n) = F(n + 1), and A(n) = B(n) 1 = F(n + 1) 1 60

Compute the nth Fibonacci Number Iteratively Algorithm Fib(n) //Computes the nth Fibonacci number iteratively by using its definition //Input: A nonnegative integer n //Output: The nth Fibonacci number F[0] 0; F[1] 1 for i 2 to n do F[i] F[i 1] + F[i 2] return F[n] This algorithm clearly makes n 1 additions. Hence, A(n) (n). 61

(log n) Algorithm to Compute the nth Fibonacci Number It is based on the formula 62