1.2 Basic Concepts
What is Performance Analysis? Performance Analysis Space Complexity: - the amount of memory space used by the algorithm Time Complexity - the amount of computing time used by the algorithm Typically, the more (less) space & the less (more) time are desirable! But, sometimes we need to trade off space vs. time. 2
Space complexity Examples A total sum of numbers. Space =? float sum (float list[ ], int n) { float tempsum = 0; int i; for (i = 0; i < n; i++) tempsum += list[i]; return tempsum; list n 6 9 1 8 4 7 An addition of two n x m matrices. Space =? Representing an n x n sparse matrix. Space =? 3
Time Complexity (1) Example: Sum of Integers less than 10 6 int sum = 0; for (i = 0; i < 1000000; i++) { sum = sum + i ; What is time complexity? Theoretical Speed: 10 6 (additions) Practical Speed: 10 msec. (Assume: Pentium III, 256M memory) Which criteria is more reasonable? Theoretical speed gives better criteria. Why? 4
Time Complexity (2) Time Complexity Criteria? Theoretical Speed - the number of operations performed by the algorithm. Practical Speed - the execution time performed by the algorithm (by using computer) Why Theoretical speed is more reasonable? No need to do Implementation (i.e., coding) H/W is not required S/W (i.e., programming language) is not required A consistent result is offered for all conditions! (Only pen and paper are used!) 5
Computation time Time Complexity (3) Properties in Time Complexity The running time of an algorithm typically grows with the input size (i.e., problem size). The running time of an algorithm is represented as a function of the input size (n) Main factors of the running time (i.e., time complexity) Linear Loops Logarithmic Loops Nested Loops Functions w.r.t. n ; f(n) for(i=1; i<n; i*=2) { for(j=0; j<n; j++) { for(i=0; i<n; i++) { for(j=0; j<n; j++) { n, Input size (problem size) 6
Time is proportional to n Addition 1 Linear Loops for (i = 1; i <= n; i++) { // application code f(n) =? Addition 2 for (i = 1; i <= n; i += 2) { // application code f(n) =? 7
Time is proportional to log 2 (n) Multiply Logarithmic Loops for (i = 1; i <= n; i *= 2) { // application code 2? n f(n) =? Division for (i = n; i >= 1; i /= 2) { // application code n/2? 1 f(n) =? 8
Nested Loops (1) Basic Formula Total Number (of Operations) = No. of outer loops * No. of inner loops Quadratic for (i = 1; i <= n; i++) { for (j = 1; j <= n; j++) { f(n) =? // application code 9
Nested Loops (2) Dependent quadratic for (i = 1; i <= n; i++) { for (j = 1; j <= i; j++) { // application code outer loop: n times inner loop: (n+1)/2 times f(n) =? Linear logarithm for (i = 1; i <= n; i++) { for (j = 1; j <= n; j *= 2) { // application code outer loop: n times inner loop: log 2 (n) times f(n) =? 10
Worst Case / Average Case Time Complexity Analysis Worst Case The worst case in terms of Input data Upper bound of the algorithm (on the problem) is offered Easier to compute it! Average Case Average case Worst case The average case in terms of all possible Input data The average complexity is offered. Difficult to compute it; we need probability values! This course mainly focuses on the worst case! 11
Exercise: Worst Case and Average Case Analyze the worst and average case for the following problem: Find X from n array (i.e., input data) that are unsorted! The n input data are stored in array, named as list. Sequential search algorithm is used. The running time depends on the number of comparisons. By the total probability theorem: Worst case! n Worst Case: W(n) list 6 9 1 4 8 X X exists at the last position of list or X does not exist. W(n) = max{max{1, 2,, n, n Average Case: A(n) q: the probability that X exists in list list list 6 6 9 9 1 X 4 4 8 8 2 2 T(i): no. of comparisons when X exists at the ith index of list T(1)+T(2)+ +T(n) A(n) = q (1/n) (1 + 2 + + n) + (1 q) n = (q/2) (n + 1) + (1 q) n 12
Comparing Time Complexities (1) An Observation on Algorithms Many algorithms exist for solving a given problem! Important to know faster one by comparing the time performance! What are the criteria for the time performance? Ex) problem g(n) algorithm A algorithm B f(n) f(n) = 100n g(n) = n 2 Question: Which one is faster/better? 100 n 13
Comparing Time Complexities (2) Ex) Please analyze the time performance on FindSum! No. of Operations function FindSum (float list[], int n) { float sum = 0; int i; for (i = 0; i < n; i++) sum += list[i]; return sum; 0 1 0 n + 1 n 1 Total : 2n + 3 14
Comparing Time Complexities (3) Algorithm FindSum executes 2n+3 operations in the worst case. We define a : time taken by the fastest operation b : time taken by the slowest operation Let T(n) be worst case time of FindSum. Then, a (2n + 3) T(n) b (2n + 3) Thus, the running time T(n) is bounded by two linear functions. 15
Big-Oh Notation: O( ) Given functions f(n) and g(n), we say that f(n) = O(g(n)) if there are positive constants c and n 0 such that f(n) c g(n) for all n n 0 time c g(n) f(n) n 0 A break even point n 0 always exists without regard to c! input size n The constant c depends on H/W or S/W environments, and it does not affect the growth rate of complexity If n is greater than n 0 (i.e., input size is large enough), g(n) is greater than f(n) all the time. 16
Exercise: Big Oh(O) Notation Ex 1: 100n = O(n 2 )? 100n c n 2 n(cn 100) 0 pick c = 1 and n 0 = 100 1 n 2 100n Ex 2: 7n 2 = O(n)? 100 n We need c and n 0 such that 7n 2 c n for n n 0 pick c = 7 and n 0 = 1 Ex 3: 3n 3 + 20n 2 + 5 = O(n 3 )? We need c and n 0 such that 3n 3 + 20n 2 + 5 c n 3 for n n 0 pick c = 4 and n 0 = 21 Ex 4: 3n 2 = O(100n)? c and n 0 that satisfy 3n 2 c 100n for n n 0 do not exist! 17
Big Oh and Growth Rate Big Oh notation gives an upper bound on the growth rate of a function f(n)= O(g(n)) means that growth rate of f(n) is no more than the growth rate of g(n). f(n)= O(g(n)) g(n)= O(f(n)) ` f(n) g(n) g(n) grows faster f(n) grows faster yes no no yes f(n) the same yes yes n 18
Big Oh Rules If f(n) = a k n k +... + a 1 n + a 0, (k > 0), then f(n) = O(n k ) Proof) f(n) a k n k + a k-1 n k-1 +...+ a 1 n + a 0 = { a k + a k-1 /n +...+ a 1 /n k-1 + a 0 /n k n k { a k + a k-1 +...+ a 1 + a 0 n k = c n k, where c = a k + a k-1 +...+ a 1 + a 0 = O(n k ) Big Oh Rules (1) Drop lower order terms (2) Drop constant factors Examples: 100n 2 + 2n = O(n 2 ) 2n 3 + 10n 2 + 40 = O(n 3 ) 19
Class of Time Complexities Polynomial Time Constant : O(1) Ex: 30, 1, 100,.. Logarithmic : O(log 2 (n)) Ex: 2log 2 n, log 2 n + 10,... Linear : O(n) Ex: n, 100n, 5n, 100n + 2log 2 n + 3,... Linear Logarithmic : O(nlog 2 (n)) Ex: 2nlog 2 n, 2nlog 2 n + n + 5,... Square : O(n 2 ) Ex: n 2, 10n 2, 2n 2 + 2nlog 2 n + 5,... Cubic : O(n 3 ) Ex: n 3, 2n 3, 4n 3 + 8n 3 + 7n 2 + 5,............. Exponential Time O(2 n ) O(n!) O(n n ) 20
Summary of Time Complexities Time Description What if n doubles? O(1) independent of input size constant O(log 2 n) slightly increase as n grows grow with a constant O(n) linearly increase w.r.t. n increase double O(n log 2 n) not bad even if n is large! grow more than double O(n 2 ) reasonable if n is small (but bad)! increase 4 times O(2 n ) Nonrealistic! increase exponentially 21
Function Values log n n n log n n 2 n 3 2 n 0 1 0 1 1 2 1 2 2 4 8 4 2 4 8 16 64 16 3 8 24 64 512 256 4 16 64 256 4,096 65,536 5 32 160 1024 32,768 4,294,967,296 22
Exercise (1) What is the time complexity of the following algorithm? for ( int i = 0, i < n; i++ ) for ( int j = 0; j < i; j++ ) for ( int k = 0; k < j; k++ ) sum ++; T(n) = i=1, n ( j=1, i ( k=1, j 1 ) ) = i=1, n ( j=1, i (j) ) = i=1, n (i(i+1)/2) = (1/2) i=1, n (i 2 +i) = (1/2){(n(n+1)(2n+1)/6 + (n(n+1))/2 = O(n 3 ) 23
Exercise (2) What is the time complexity of GCD algorithm? function GCD (int L, int S) int R; while (S > 0) { R = L % S; L = S; S = R; return (L); Ex) Find the GCD of L=50, S=30. R = L%S; R = 50%30 = 20; L = S; L = 30; S = R; S = 20; ------------------------------------ R = L%S; R = 30%20 = 10; L = S; L = 20; S = R; S = 10; ------------------------------------ R = L%S; R = 20%10 = 0; L = S; L = 10; S = R; S = 0; Hint) It is impossible that the remainder R is greater than ½ L if L is divided by S (L>S). For instance, if L=50 and S=30, then R=? 24