C H A P T E R 3 Principles of Algorithm Analysis 3.1 Computer Programs The design of computer programs requires:- 1. An algorithm that is easy to understand, code and debug. This is the concern of software engineering. 2. An algorithm that makes efficient use of the computer s resources. This is the concern of data structure and algorithm analysis. Two algorithms, A and B, for solving a given problem. Let the running times of the algorithms to be T a (n) and T b (n) Suppose the problem size is n 0 and T a (n 0 ) < T b (n 0 ). Then algorithm A is better than algorithm B for problem size n 0 only, which is not good enough If T a (n) < T b (n) for all n, n n 0, then algorithm A is better than algorithm B regardless of the problem size. Note: Our primarily concern is to estimate the time of an algorithm instead of computing its exact time. A useful measurement of time can be obtained by counting the fundamental, dominating steps of the algorithm. E.g. counting the number of comparisons in a sorting algorithm Page 1 of 6
3.2 Running Time Let - c op = execution time of an algorithm s basic operation on a computer. - C(n) = number of times the basic operation is executed for this algorithm - T(n) = running time of this algorithm on the computer. T(n) c op C(n) Assume that for an algorithm, the operation count :- C(n) = ½ n(n-1) ½ n 2 How much longer will the algorithm run if the input size is doubled? 2 T 2n c (2) 1 (2) opc n n 2 4 1 2 T ()() n copc n () n 2 The run time will be 4 times longer when the input size is doubled. For algorithm analysis, we emphasize on the operation count s order of growth for large input sizes. Small input size cannot distinguish efficient algorithm from inefficient ones. Three notations are defined to compare and rank the order of growth ( the efficiency) of different algorithms. They are :- O (Big Oh), (Big Omega) and (Big Theta) 3.3 Asymptotic Notation When we look at input sizes large enough to make only the order of growth of the running time relevant, we are studying the asymptotic efficiency of algorithms. Usually, an algorithm that is asymptotically more efficient will be the best choice for all but very small inputs. 3.31 The Big Oh Notation (O) - A notation used for characterizing the asymptotic behavior of functions. - Gives an asymptotic upper bound for a function to within a constant factor. Definition :- Given non-negative functions f(n) and g(n), we say that f(n) = O (g(n)), if there exists an integer n 0, and a constant k>0 such that f(n) kg(n) for all integers nn 0. f(n) = O(g(n)) :- f(n) is of order at most g(n) or f(n) is big oh of g(n) The value of n 0 is the minimum possible value. Page 2 of 6
Example 1: 2n + 10 is O(n) 100 2n 10 kn ( k - 2) n 10 10 n k 2 Pick k = 3 and therefore n 0 = 10. 10 Example 2: Given f(n) = 8n + 128 and g(n) is n 2. Is f(n) = O(g(n))? We need to find an integer n 0 and a constant k > 0 such that for all integers n n 0, f(n) kn 2. 1 3n 2n+10 10 100 Therefore, 8n + 128 kn 2 kn 2 8n 128 0 If we set k=1, we have (n+8) (n-16) 0 Hence, (n-16) 0, n 0 = 16 when k=1. Example 3: To prove that the function n 2 is NOT O(n). n 2 kn n k The equality cannot be satisfied as k must be a constant. Example 4: 7n-2 7n-2 is O(n) We need k > 0 and n 0 1, such that 7n-2 kn for n n 0 7n-2 kn Assume k = 7, n o =1. Example 5: 3n 3 + 20n 2 + 5 3n 3 + 20n 2 + 5 is O(n 3 ) We need k > 0 and n 0 1, such that 3n 3 + 20n 2 + 5 kn 3 for n n 0 3n 3 + 20n 2 + 5 kn 3 Assume k = 4 20n 2 + 5 4n 3 3n 3 20n 2 + 5 n 2 n n o = 21. Page 3 of 6
Example 6: 3lg (n) + 5 3lg(n) + 5 is O(lg(n)) We need k > 0 and n 0 1, such that 3lg(n) + 5 k lg(n) for n n 0 3lg(n) + 5 k lg(n) 5 k lg(n) 3lg(n) 5 (k 3) lg(n) Analyzing the two log 2 graphs below, we can find some similarities:- 1. Both 3lg(n) and lg(n) cuts the n-axis at n=1 2. At n=2 (which is the base of the log), the graph will have a y-values equivalent to the magnitude. i.e. 3lg(n) has a value of 3 when n=2. 3 lg n lg n Therefore, to satisfy 5 (k 3) lg(n), (k 3) should be at least 5. Assume k=8, n 0 will be equal to 2. 3.3.2 Big-Oh Rules If f(n) is a polynomial of degree d, then f(n) is O(n d ), i.e. 1. Drop Lower-order terms 2. Drop Constant factors E.g. 100n + 2 is O(n) Page 4 of 6
3.3.3 Big-Oh and Growth Rate The Big-Oh notation gives an upper bound (or worst case) on the growth rate of the function. The statement f(n) is O(g(n)) means that the growth rate of f(n) is no more than the growth rate of g(n). We can use the Big-Oh to rank functions according to their growth rate. 3.3.4 Functions representing Growth Rates The amount of time required to execute an algorithm usually depends on the input size, n. Here are seven functions that often appear in algorithm analysis in ascending order in terms of running time. - Constant 1 Fastest - Logarithmic log n - Linear n - N-Log-N nlogn - Quadratic n 2 - Cubic n 3 - Exponential 2 n Slowest 3.3.5 The Big Omega Notation () The Ω-notation provides an asymptotic lower bound. f(n) = (g(n)) f(n) is of order at least g(n) or f(n) is the omega of g(n) if there exists constants k 2 and n 2 such that f(n) k 2 g(n) for all n n 2 3.3.6 The Big Theta Notation () The -notation provides an asymptotic tight bound. f(n) = (g(n)) f(n) is of order g(n) or f(n) is theta of g(n) if f(n)=o (g(n)) and f(n)= (g(n)) Page 5 of 6
Example 7: Given f(n) = 60n 2 + 5n + 1, g(n) = n 2, prove that 60n 2 +5n+1=(n 2 ) 60n 2 + 5n + 1 < 66n 2 for all n 1 Hence we can take k 1 =66 and n 1 =1, and conclude that f(n)=o(g(n 2 )) Since, 60n 2 + 5n + 1 > 60n 2 for all n 1 we can take k 2 =60 and n 2 =1, and conclude that f(n)=(g(n 2 )) Based on the above, we have 60n 2 + 5n + 1 = (n 2 ) Example 8: Given f(n) = 2n + 3lg(n), g(n) = n, prove that 2n + 3lg(n)=(n) Proof:- 60n 2 + 5n + 1 < 60n 2 + 5n 2 + n 2 Proof:- 2n + 3lg(n) 2n + 3n 2n + 3lg(n) 5n for all n 1 Hence we can take k 1 =5 and n 1 =1, and conclude that f(n)=o(g(n)) Since, 2n + 3lg(n) 2n for all n 1 we can take k 2 =2 and n 2 =1, and conclude that f(n)=(g(n)) Based on the above, we have 2n + 3lg(n) = (n) Page 6 of 6