Analysis of Algorithms The Time Complexity of an Algorithm Specifies how the running time depends on the size of the input.
Purpose To estimate how long a program will run. To estimate the largest input that can reasonably be given to the program. To compare the efficiency of different algorithms. To help focus on the parts of code that are executed the largest number of times. To choose an algorithm for an application. 2
Time Complexity Is a Function Specifies how the running time depends on the size of the input. A function mapping size of input time T(n) executed. 3
Example: Insertion Sort 4
Example: Insertion Sort 5
Example: Insertion Sort nn ( + 1) n 2 Worst case (reverse order): tj = j : j = 1 T( n) θ( n ) j = 2 2 6
Example: Merge Sort 7
8
Example: Merge Sort 9
Example: Merge Sort 10 Tn ( ) θ( nlog n)
Relevant Mathematics: Classifying Functions Time Input Size
Assumed Knowledge Chapter 22. Existential and Universal Quantifiers g b Loves( b, g) g b Loves( b, g) Chapter 24. Logarithms and Exponentials 12
Classifying Functions Describing how fast a function grows without going into too much detail.
Which are more alike? 14
Which are more alike? Mammals 15
Which are more alike? 16
Which are more alike? Dogs 17
Classifying Animals Vertebrates Fish Reptiles Mammals Birds Dogs Giraffe 18
Classifying Functions n T(n) 10 100 1,000 10,000 log n 3 6 9 13 n 1/2 3 10 31 100 n 10 100 1,000 10,000 n log n 30 600 9,000 130,000 n 2 100 10,000 10 6 10 8 n 3 1,000 10 6 10 9 10 12 2 n 1,024 10 30 10 300 10 3000 Note: The universe is estimated to contain ~10 80 particles. 19
Which are more alike? n 1000 n 2 2 n 20
Which are more alike? n 1000 n 2 2 n Polynomials 21
Which are more alike? 1000n 2 3n 2 2n 3 22
Which are more alike? 1000n 2 3n 2 2n 3 Quadratic 23
Classifying Functions? Functions 24
Classifying Functions Functions Double Exponential Exp Exponential Polynomial Poly Logarithmic Logarithmic Constant 5 5 log n (log n) 5 n 5 2 5n 2 n5 25n 2 25
Classifying Functions? Polynomial 26
Classifying Functions Polynomial Linear Quadratic Cubic 4 th Order 5n 5n 2 5n 3 5n 4 27
Classifying Functions Functions Double Exponential Exp Exponential Polynomial Poly Logarithmic Logarithmic Constant 5 5 log n (log n) 5 n 5 2 5n 2 n5 25n 2 28
Properties of the Logarithm 1 Changing base: s logan = log log a b b n Swapping base and argum ent: loga 1 b = log b a 29
Logarithmic log 10 n = # digits to write n log 2 n = # bits to write n = 3.32 log 10 n log(n 1000 ) = 1000 log(n) Differ only by a multiplicative constant. 1 Changing base: s logan = log log a b b n 30
Classifying Functions Functions Double Exponential Exp Exponential Polynomial Poly Logarithmic Logarithmic Constant 5 5 log n (log n) 5 n 5 2 5n 2 n5 25n 2 31
Poly Logarithmic (log n) 5 = log 5 n 32
(Poly)Logarithmic << Polynomial log 1000 n << n 0.001 For sufficiently large n 33
Classifying Functions Polynomial Linear Quadratic Cubic 4 th Order 5n 5n 2 5n 3 5n 4 34
Linear << Quadratic 10000 n << 0.0001 n 2 For sufficiently large n 35
Classifying Functions Functions Double Exponential Exp Exponential Polynomial Poly Logarithmic Logarithmic Constant 5 5 log n (log n) 5 n 5 2 5n 2 n5 25n 2 36
Polynomial << Exponential n 1000 << 2 0.001 n For sufficiently large n 37
Ordering Functions Functions Constant Logarithmic Poly Logarithmic Polynomial Exponential Exp << << << << << << Double Exponential 5 << 5 log n << (log n) 5 << n 5 << 2 5n << 2 << 2 n5 25n 38
Which Functions are Constant? Yes Yes Yes Yes Yes No 5 1,000,000,000,000 0.0000000000001-5 0 8 + sin(n) 39
Which Functions are Constant? The running time of the algorithm is a Constant It does not depend significantly on the size of the input. Yes Yes Yes No No Yes 5 1,000,000,000,000 0.0000000000001-5 0 8 + sin(n) 9 7 Lies in between 40
Which Functions are Quadratic? n 2? 41
Which Functions are Quadratic? n 2 0.001 n 2 1000 n 2 Some constant times n 2. 42
Which Functions are Quadratic? n 2 0.001 n 2 1000 n 2 5n 2 + 3n + 2log n 43
Which Functions are Quadratic? n 2 0.001 n 2 1000 n 2 5n 2 + 3n + 2log n Lies in between 44
Which Functions are Quadratic? n 2 0.001 n 2 1000 n 2 5n 2 + 3n + 2log n Ignore low-order terms Ignore multiplicative constants. Ignore "small" values of n. Write θ(n 2 ). 45
Which Functions are Polynomial? n 5? 46
Which Functions are Polynomial? n c n 0.0001 n 10000 n to some constant power. 47
Which Functions are Polynomial? n c n 0.0001 n 10000 5n 2 + 8n + 2log n 5n 2 log n 5n 2.5 48
Which Functions are Polynomial? n c n 0.0001 n 10000 5n 2 + 8n + 2log n 5n 2 log n 5n 2.5 Lie in between 49
Which Functions are Polynomials? n c n 0.0001 n 10000 5n 2 + 8n + 2log n 5n 2 log n 5n 2.5 Ignore low-order terms Ignore power constant. Ignore "small" values of n. Write n θ(1) 50
Which Functions are Exponential? 2 n? 51
Which Functions are Exponential? 2 n 2 0.0001 n 2 10000 n 2 raised to a linear function of n. 52
Which Functions are Exponential? 2 n 2 0.0001 n 2 10000 n 8 n 2 n / n 100 2 n n 100 too small? too big? 53
Which Functions are Exponential? 2 n 2 0.0001 n 2 10000 n 8 n 2 n / n 100 2 n n 100 = 2 3n > 2 0.5n < 2 2n Lie in between 54
Which Functions are Exponential? 2 n 2 0.0001 n 2 10000 n 8 n 2 n / n 100 2 n n 100 = 2 3n > 2 0.5n < 2 2n 2 0.5n > n 100 2 n = 2 0.5n 2 0.5n > n 100 2 0.5n 2 n / n 100 > 2 0.5n 55
Which Functions are Exponential? 2 n 2 0.0001 n 2 10000 n 8 n 2 n / n 100 2 n n 100 Ignore low-order terms Ignore base. Ignore "small" values of n. Ignore polynomial factors. Write 2 θ(n) 56
Classifying Functions f(n) = 8 2 4n / n 100 + 5 n 3 Tell me the most significant thing about your function Rank constants in significance. 57
Classifying Functions f(n) = 8 2 4n / n 100 + 5 n 3 Tell me the most significant thing about your function because 2 θ(n) 2 3n < f(n) < 2 4n 58
Classifying Functions f(n) = 8 2 4n / n 100 + 5 n 3 Tell me the most significant thing about your function 2 θ(n) 2 4n /n θ(1) θ(2 4n /n 100 ) 8 2 4n / n 100 + n θ(1) 8 2 4n / n 100 + θ(n 3 ) 8 2 4n / n 100 + 5 n 3 59
Notations Theta f(n) = θ(g(n)) f(n) c g(n) Big Oh f(n) = O(g(n)) f(n) c g(n) Big Omega f(n) = Ω(g(n)) f(n) c g(n) Little Oh f(n) = o(g(n)) f(n) << c g(n) Little Omega f(n) = ω(g(n)) f(n) >> c g(n) 60
Definition of Theta f(n) = θ(g(n)) ccn,, > 0: n n, cgn ( ) fn ( ) cgn ( ) 1 2 0 0 1 2 61
Definition of Theta f(n) = θ(g(n)) ccn,, > 0: n n, cgn ( ) fn ( ) cgn ( ) 1 2 0 0 1 2 f(n) is sandwiched between c 1 g(n) and c 2 g(n) 62
Definition of Theta f(n) = θ(g(n)) cc,, n > 0: n n, cgn ( ) fn ( ) cgn ( ) 1 2 0 0 1 2 f(n) is sandwiched between c 1 g(n) and c 2 g(n) for some sufficiently small c 1 (= 0.0001) for some sufficiently large c 2 (= 1000) 63
Definition of Theta f(n) = θ(g(n)) ccn,, > 0: n n, cgn ( ) fn ( ) cgn ( ) 1 2 0 0 1 2 For all sufficiently large n 64
Definition of Theta f(n) = θ(g(n)) ccn,, > 0: n n, cgn ( ) fn ( ) cgn ( ) 1 2 0 0 1 2 For all sufficiently large n For some definition of sufficiently large 65
Definition of Theta 3n 2 + 7n + 8 = θ(n 2 ) ccn,, > 0: n n, cgn ( ) fn ( ) cgn ( ) 1 2 0 0 1 2 66
Definition of Theta 3n 2 + 7n + 8 = θ(n 2 ) ccn,, > 0: n n, cgn ( ) fn ( ) cgn ( ) 1 2 0 0 1 2?? c 1 n 2 3n 2 + 7n + 8 c 2 n 2 67
Definition of Theta 3n 2 + 7n + 8 = θ(n 2 ) ccn,, > 0: n n, cgn ( ) fn ( ) cgn ( ) 1 2 0 0 1 2 3 4 3 n 2 3n 2 + 7n + 8 4 n 2 68
Definition of Theta 3n 2 + 7n + 8 = θ(n 2 ) ccn,, > 0: n n, cgn ( ) fn ( ) cgn ( ) 1 2 0 0 1 2 1 3 1 2 3 1 2 + 7 1 + 8 4 1 2 69 False
Definition of Theta 3n 2 + 7n + 8 = θ(n 2 ) cc,, n > 0: n n, cgn ( ) fn ( ) cgn ( ) 1 2 0 0 1 2 7 3 7 2 3 7 2 + 7 7 + 8 4 7 2 70 False
Definition of Theta 3n 2 + 7n + 8 = θ(n 2 ) ccn,, > 0: n n, cgn ( ) fn ( ) cgn ( ) 1 2 0 0 1 2 8 3 8 2 3 8 2 + 7 8 + 8 4 8 2 71 True
Definition of Theta 3n 2 + 7n + 8 = θ(n 2 ) cc,, n > 0: n n, cgn ( ) fn ( ) cgn ( ) 1 2 0 0 1 2 9 3 9 2 3 9 2 + 7 9 + 8 4 9 2 72 True
Definition of Theta 3n 2 + 7n + 8 = θ(n 2 ) ccn,, > 0: n n, cgn ( ) fn ( ) cgn ( ) 1 2 0 0 1 2 10 3 10 2 3 10 2 + 7 10 + 8 4 10 2 73 True
Definition of Theta 3n 2 + 7n + 8 = θ(n 2 ) ccn,, > 0: n n, cgn ( ) fn ( ) cgn ( ) 1 2 0 0 1 2? 3 n 2 3n 2 + 7n + 8 4 n 2 74
Definition of Theta 3n 2 + 7n + 8 = θ(n 2 ) ccn,, > 0: n n, cgn ( ) fn ( ) cgn ( ) 1 2 0 0 1 2 8 n 8 3 n 2 3n 2 + 7n + 8 4 n 2 75
Definition of Theta 3n 2 + 7n + 8 = θ(n 2 ) ccn,, > 0: n n, cgn ( ) fn ( ) cgn ( ) 1 2 0 0 1 2 n 8 3 n 2 3n 2 + 7n + 8 4 n 2 76 True
Definition of Theta 3n 2 + 7n + 8 = θ(n 2 ) ccn,, > 0: n n, cgn ( ) fn ( ) cgn ( ) 1 2 0 0 1 2 n 8 3 n 2 3n 2 + 7n + 8 4 n 2 77 7n + 8 1 n 2 7 + 8 / n 1 n True
Definition of Theta 3n 2 + 7n + 8 = θ(n 2 ) True ccn,, > 0: n n, cgn ( ) fn ( ) cgn ( ) 1 2 0 0 1 2 3 4 8 n 8 3 n 2 3n 2 + 7n + 8 4 n 2 78
Asymptotic Notation Theta f(n) = θ(g(n)) f(n) c g(n) BigOh f(n) = O(g(n)) f(n) c g(n) Omega f(n) = Ω(g(n)) f(n) c g(n) Little Oh f(n) = o(g(n)) f(n) << c g(n) Little Omega f(n) = ω(g(n)) f(n) >> c g(n) 79
Definition of Theta 3n 2 + 7n + 8 = θ(n) ccn,, > 0: n n, cgn ( ) fn ( ) cgn ( ) 1 2 0 0 1 2 80
Definition of Theta 3n 2 + 7n + 8 = θ(n) ccn,, > 0: n n, cgn ( ) fn ( ) cgn ( ) 1 2 0 0 1 2?? c 1 n 3n 2 + 7n + 8 c 2 n 81
Definition of Theta 3n 2 + 7n + 8 = θ(n) ccn,, > 0: n n, cgn ( ) fn ( ) cgn ( ) 1 2 0 0 1 2 3 100 3 n 3n 2 + 7n + 8 100 n 82
Definition of Theta 3n 2 + 7n + 8 = θ(n) ccn,, > 0: n n, cgn ( ) fn ( ) cgn ( ) 1 2 0 0 1 2? 3 n 3n 2 + 7n + 8 100 n 83
Definition of Theta 3n 2 + 7n + 8 = θ(n) ccn,, > 0: n n, cgn ( ) fn ( ) cgn ( ) 1 2 0 0 1 2 100 3 100 3 100 2 + 7 100 + 8 100 100 84 False
Definition of Theta 3n 2 + 7n + 8 = θ(n) ccn,, > 0: n n, cgn ( ) fn ( ) cgn ( ) 1 2 0 0 1 2 3 10,000 3 n 3n 2 + 7n + 8 10,000 n 85
Definition of Theta 3n 2 + 7n + 8 = θ(n) ccn,, > 0: n n, cgn ( ) fn ( ) cgn ( ) 1 2 0 0 1 2 10,000 3 10,000 3 10,000 2 + 7 10,000 + 8 10,000 10,000 86 False
Definition of Theta 3n 2 + 7n + 8 = θ(n) What is the reverse statement? ccn,, > 0: n n, cgn ( ) fn ( ) cgn ( ) 1 2 0 0 1 2 87
Understand Quantifiers!!! b, Loves(b, MM) [ b, Loves(b, MM)] [ b, Loves(b, MM)] b, Loves(b, MM) Sam Mary Sam Mary Bob Beth Bob Beth John Marilyn Monroe John Marilyn Monroe Fred Ann Fred Ann 88
Definition of Theta 3n 2 + 7n + 8 = θ(n) The reverse statement ccn,, > 0: n n, cgn ( ) fn ( ) cgn ( ) 1 2 0 0 1 2 ccn,, > 0: n n : fn ( ) < cgn ( ) or fn ( ) > cgn ( ) 1 2 0 0 1 2 89
Definition of Theta 3n 2 + 7n + 8 = θ(n) ccn,, > 0: n n : fn ( ) < cgn ( ) or fn ( ) > cgn ( ) 1 2 0 0 1 2? 2 3 7 8 n + n + > cn 7 8 c2 3 + + > 2 n n n Satisfied for n = max( n, c ) 2 0 2 90
Order of Quantifiers f(n) = θ(g(n)) ccn,, > 0: n n, cgn ( ) fn ( ) cgn ( ) 1 2 0 0 1 2 n > 0: n n, c, c : c g( n) f( n) c g( n) 0 0 1 2 1 2? 91
Understand Quantifiers!!! g, b, loves( b, g) b, g, loves( b, g) One girl Could be a separate girl for each boy. Sam Mary Sam Mary Bob Beth Bob Beth John Marilin Monro John Marilin Monro Fred Ann Fred Ann 92
Order of Quantifiers f(n) = θ(g(n)) ccn,, > 0: n n, cgn ( ) fn ( ) cgn ( ) 1 2 0 0 1 2 n > 0: n n, c, c : c g( n) f( n) c g( n) 0 0 1 2 1 2 No! It cannot be a different c 1 and c 2 for each n. 93
Other Notations Theta f(n) = θ(g(n)) f(n) c g(n) BigOh f(n) = O(g(n)) f(n) c g(n) Omega f(n) = Ω(g(n)) f(n) c g(n) Little Oh f(n) = o(g(n)) f(n) << c g(n) Little Omega f(n) = ω(g(n)) f(n) >> c g(n) 94
Definition of Big Omega fn ( ) cg( n) fn ( ) Ω( gn ( )) gn ( ) > 0:, ( ) ( ) cn, 0 n n0 f n cgn n 95
Definition of Big Oh cg( n) fn ( ) fn ( ) Ogn ( ( )) gn ( ) > 0:, ( ) ( ) cn, 0 n n0 f n cgn n 96
Definition of Little Omega fn ( ) cg( n) fn ( ) ω( gn ( )) gn ( ) c > 0, n > 0 : n n, f( n) > cg( n) 0 0 n 97
Definition of Little Oh cg( n) fn ( ) fn ( ) ogn ( ( )) gn ( ) c > 0, n > 0 : n n, f( n) < cg( n) 0 0 n 98
Time Complexity Is a Function Specifies how the running time depends on the size of the input. A function mapping size of input time T(n) executed. 99
Definition of Time? 100
Definition of Time # of seconds (machine dependent). # lines of code executed. # of times a specific operation is performed addition). (e.g., These are all reasonable definitions of time, because they are within a constant factor of each other. 101
Definition of Input Size A good definition: #bits Examples: 1. Sorting an array A of k-bit e.g., k=16. Input size n = k length( A) integers. Note that, since n length(a), sufficient to define n = length( A). 2. Multiplying A,B. Let n = #bits used to represent A A Let n = B #bits used to represent B Then input size n = na + nb Note that na ; log 2A and nb ; log2b T hus input size n ; log A + log B 2 2 102
Time Complexity Is a Function Specifies how the running time depends on the size of the input. A function mapping: size of input time T(n) executed. Or more precisely: # of bits n needed to represent the input # of operations T(n) executed. 103
Time Complexity of an Algorithm The time complexity of an algorithm is the largest time required on any input of size n. (Worst case analysis.) O(n 2 ): For any input size n>n 0, the algorithm takes no more than cn 2 time on every input. Ω(n 2 ): For any input size n>n 0, the algorithm takes at least cn 2 time on at least one input. θ (n 2 ): Do both. 104
What is the height of tallest person in the class? Bigger than this? Smaller than this? Need to find only one person who is taller Need to look at every person 105
Time Complexity of a Problem The time complexity of a problem is the time complexity of the fastest algorithm that solves the problem. O(n 2 ): Provide an algorithm that solves the problem in no more than this time. Remember: for every input, i.e. worst case analysis! Ω(n 2 ): Prove that no algorithm can solve it faster. Remember: only need one input that takes at least this long! θ (n 2 ): Do both. 106