CSE 3101Z Design and Analysis of Algorithms The Time Complexity of an Algorithm Specifies how the running time depends on the size of the input.
Purpose To estimate how long a program will run. To estimate the largest input that can reasonably be given to the program. To compare the efficiency of different algorithms. To help focus on the parts of code that are executed the largest number of times. To choose an algorithm for an application. COSC 3101, PROF. J. ELDER 2
Time Complexity Is a Function Specifies how the running time depends on the size of the input. A function mapping size of input executed. time T(n) COSC 3101, PROF. J. ELDER 3
Example: Insertion Sort COSC 3101, PROF. J. ELDER 4
Example: Insertion Sort COSC 3101, PROF. J. ELDER 5
Example: Insertion Sort nn ( + 1) n 2 Worst case (reverse order): tj = j : j = 1 T( n) θ( n ) j = 2 2 COSC 3101, PROF. J. ELDER 6
Example: Merge Sort COSC 3101, PROF. J. ELDER 7
COSC 3101, PROF. J. ELDER 8
Example: Merge Sort COSC 3101, PROF. J. ELDER 9
Example: Merge Sort COSC 3101, PROF. J. ELDER 10 Tn ( ) θ( nlog n)
Relevant Mathematics: Classifying Functions Time Input Size
Assumed Knowledge See J. Edmonds, How to Think About Algorithms (HTA): http://www.cse.yorku.ca/~jeff/courses/3101/ Chapter 22. Existential and Universal Quantifiers g b Loves( b, g) g b Loves( b, g) Chapter 24. Logarithms and Exponentials COSC 3101, PROF. J. ELDER 12
Classifying Functions Describing how fast a function grows without going into too much detail.
Which are more alike? COSC 3101, PROF. J. ELDER 14
Which are more alike? Mammals COSC 3101, PROF. J. ELDER 15
Which are more alike? COSC 3101, PROF. J. ELDER 16
Which are more alike? Dogs COSC 3101, PROF. J. ELDER 17
Fish Classifying Animals Vertebrates Reptiles Mammals Birds Dogs COSC 3101, PROF. J. ELDER 18 Giraffe
Classifying Functions n T(n) 10 100 1,000 10,000 log n 3 6 9 13 n 1/2 3 10 31 100 n 10 100 1,000 10,000 n log n 30 600 9,000 130,000 n 2 100 10,000 10 6 10 8 n 3 1,000 10 6 10 9 10 12 2 n 1,024 10 30 10 300 10 3000 Note: The universe is estimated to contain ~10 80 particles. COSC 3101, PROF. J. ELDER 19
Which are more alike? n 1000 n 2 2 n COSC 3101, PROF. J. ELDER 20
End of Lecture 1 Wed, March 4, 2009
Which are more alike? n 1000 n 2 2 n Polynomials COSC 3101, PROF. J. ELDER 22
Which are more alike? 1000n 2 3n 2 2n 3 COSC 3101, PROF. J. ELDER 23
Which are more alike? 1000n 2 3n 2 2n 3 Quadratic COSC 3101, PROF. J. ELDER 24
Classifying Functions? Functions COSC 3101, PROF. J. ELDER 25
Classifying Functions Functions Double Exponential Exp Exponential Polynomial Poly Logarithmic Logarithmic Constant 5 5 log n (log n) 5 n 5 2 5n 2 n5 2 2 5n COSC 3101, PROF. J. ELDER 26
Classifying Functions? Polynomial COSC 3101, PROF. J. ELDER 27
Classifying Functions Polynomial Linear Quadratic Cubic 4 th Order 5n 5n 2 5n 3 5n 4 COSC 3101, PROF. J. ELDER 28
Classifying Functions Functions Double Exponential Exp Exponential Polynomial Poly Logarithmic Logarithmic Constant 5 5 log n (log n) 5 n 5 2 5n 2 n5 2 2 5n COSC 3101, PROF. J. ELDER 29
Properties of the Logarithm Changing base s: log a n 1 = log log a b b n Swapping base and argum ent: loga 1 b = log b a COSC 3101, PROF. J. ELDER 30
Logarithmic log 10 n = # digits to write n log 2 n = # bits to write n = 3.32 log 10 n log(n 1000 ) = 1000 log(n) Differ only by a multiplicative constant. Changing base s: log a 1 n = log log a b b n COSC 3101, PROF. J. ELDER 31
Classifying Functions Functions Double Exponential Exp Exponential Polynomial Poly Logarithmic Logarithmic Constant (log n) 5 n 5 2 2 5n 5 5 log n n 5 2 5n 2 COSC 3101, PROF. J. ELDER 32
Poly Logarithmic (log n) 5 = log 5 n COSC 3101, PROF. J. ELDER 33
(Poly)Logarithmic << Polynomial log 1000 n << n 0.001 For sufficiently large n COSC 3101, PROF. J. ELDER 34
Classifying Functions Polynomial Linear Quadratic Cubic 4 th Order 5n 5n 2 5n 3 5n 4 COSC 3101, PROF. J. ELDER 35
Linear << Quadratic 10000 n << 0.0001 n 2 For sufficiently large n COSC 3101, PROF. J. ELDER 36
Classifying Functions Functions Double Exponential Exp Exponential Polynomial Poly Logarithmic Logarithmic Constant 5 5 log n (log n) 5 n 5 2 5n 2 n5 2 2 5n COSC 3101, PROF. J. ELDER 37
Polynomial << Exponential n 1000 << 2 0.001 n For sufficiently large n COSC 3101, PROF. J. ELDER 38
Ordering Functions Functions Double Exponential Exp << << << << << << Exponential Polynomial Poly Logarithmic Logarithmic Constant 5 5 log n << (log n) 5 << n 5 2 5n 2 n5 2 << 2 5n << << << COSC 3101, PROF. J. ELDER 39
Which Functions are Constant? Yes Yes Yes Yes Yes No 5 1,000,000,000,000 0.0000000000001-5 0 8 + sin(n) COSC 3101, PROF. J. ELDER 40
Which Functions are Constant? The running time of the algorithm is a Constant It does not depend significantly on the size of the input. Yes Yes Yes No No Yes 5 1,000,000,000,000 0.0000000000001-5 0 8 + sin(n) 9 7 Lies in between COSC 3101, PROF. J. ELDER 41
Which Functions are Quadratic? n 2? COSC 3101, PROF. J. ELDER 42
Which Functions are Quadratic? n 2 0.001 n 2 1000 n 2 Some constant times n 2. COSC 3101, PROF. J. ELDER 43
Which Functions are Quadratic? n 2 0.001 n 2 1000 n 2 5n 2 + 3n + 2log n COSC 3101, PROF. J. ELDER 44
Which Functions are Quadratic? n 2 0.001 n 2 1000 n 2 Lies in between 5n 2 + 3n + 2log n COSC 3101, PROF. J. ELDER 45
Which Functions are Quadratic? n 2 0.001 n 2 1000 n 2 5n 2 + 3n + 2log n Ignore low-order terms Ignore multiplicative constants. Ignore "small" values of n. Write θ(n 2 ). COSC 3101, PROF. J. ELDER 46
Which Functions are Polynomial? n 5? COSC 3101, PROF. J. ELDER 47
Which Functions are Polynomial? n c n 0.0001 n 10000 n to some constant power. COSC 3101, PROF. J. ELDER 48
Which Functions are Polynomial? n c n 0.0001 n 10000 5n 2 + 8n + 2log n 5n 2 log n 5n 2.5 COSC 3101, PROF. J. ELDER 49
Which Functions are Polynomial? n c n 0.0001 n 10000 5n 2 + 8n + 2log n 5n 2 log n 5n 2.5 Lie in between COSC 3101, PROF. J. ELDER 50
Which Functions are Polynomials? n c n 0.0001 n 10000 5n 2 + 8n + 2log n 5n 2 log n 5n 2.5 Ignore low-order terms Ignore power constant. Ignore "small" values of n. Write n θ(1) COSC 3101, PROF. J. ELDER 51
Which Functions are Exponential? 2 n? COSC 3101, PROF. J. ELDER 52
Which Functions are Exponential? 2 n 2 0.0001 n 2 10000 n 2 raised to a linear function of n. COSC 3101, PROF. J. ELDER 53
Which Functions are Exponential? 2 n 2 0.0001 n 2 10000 n 8 n 2 n / n 100 too small? 2 n n 100 too big? COSC 3101, PROF. J. ELDER 54
Which Functions are Exponential? 2 n 2 0.0001 n 2 10000 n 8 n 2 n / n 100 2 n n 100 = 2 3n > 2 0.5n < 2 2n Lie in between COSC 3101, PROF. J. ELDER 55
Which Functions are Exponential? 2 n 2 0.0001 n 2 10000 n 8 n 2 n / n 100 2 n n 100 = 2 3n > 2 0.5n < 2 2n 2 0.5n > n 100 2 n = 2 0.5n 2 0.5n > n 100 2 0.5n 2 n / n 100 > 2 0.5n COSC 3101, PROF. J. ELDER 56
Which Functions are Exponential? 2 n 2 0.0001 n 2 10000 n 8 n 2 n / n 100 2 n n 100 Ignore low-order terms Ignore base. Ignore "small" values of n. Ignore polynomial factors. Write 2 θ(n) COSC 3101, PROF. J. ELDER 57
Classifying Functions f(n) = 8 2 4n / n 100 + 5 n 3 Tell me the most significant thing about your function Rank constants in significance. COSC 3101, PROF. J. ELDER 58
Classifying Functions f(n) = 8 2 4n / n 100 + 5 n 3 Tell me the most significant thing about your function because 2 θ(n) 2 3n < f(n) < 2 4n COSC 3101, PROF. J. ELDER 59
Classifying Functions f(n) = 8 2 4n / n 100 + 5 n 3 Tell me the most significant thing about your function COSC 3101, PROF. J. ELDER 60 2 θ(n) 2 4n /n θ(1) θ(2 4n /n 100 ) 8 2 4n / n 100 + n θ(1) 8 2 4n / n 100 + θ(n 3 ) 8 2 4n / n 100 + 5 n 3
Notations Theta f(n) = θ(g(n)) f(n) c g(n) Big Oh f(n) = O(g(n)) f(n) c g(n) Big Omega f(n) = Ω(g(n)) f(n) c g(n) Little Oh f(n) = o(g(n)) f(n) << c g(n) Little Omega f(n) = ω(g(n)) f(n) >> c g(n) COSC 3101, PROF. J. ELDER 61
Definition of Theta f(n) = θ(g(n)) c, c, n > 0: n n, c g( n) f( n) c g( n) 1 2 0 0 1 2 COSC 3101, PROF. J. ELDER 62
Definition of Theta f(n) = θ(g(n)) c, c, n > 0: n n, c g( n) f( n) c g( n) 1 2 0 0 1 2 f(n) is sandwiched between c 1 g(n) and c 2 g(n) COSC 3101, PROF. J. ELDER 63
Definition of Theta f(n) = θ(g(n)) c, c, n > 0: n n, c g( n) f( n) c g( n) 1 2 0 0 1 2 f(n) is sandwiched between c 1 g(n) and c 2 g(n) for some sufficiently small c 1 (= 0.0001) for some sufficiently large c 2 (= 1000) COSC 3101, PROF. J. ELDER 64
Definition of Theta f(n) = θ(g(n)) c, c, n > 0: n n, c g( n) f( n) c g( n) 1 2 0 0 1 2 For all sufficiently large n COSC 3101, PROF. J. ELDER 65
Definition of Theta f(n) = θ(g(n)) c, c, n > 0: n n, c g( n) f( n) c g( n) 1 2 0 0 1 2 For all sufficiently large n For some definition of sufficiently large COSC 3101, PROF. J. ELDER 66
Definition of Theta 3n 2 + 7n + 8 = θ(n 2 ) c, c, n > 0: n n, c g( n) f( n) c g( n) 1 2 0 0 1 2 COSC 3101, PROF. J. ELDER 67
Definition of Theta 3n 2 + 7n + 8 = θ(n 2 ) c, c, n > 0: n n, c g( n) f( n) c g( n) 1 2 0 0 1 2?? c 1 n 2 3n 2 + 7n + 8 c 2 n 2 COSC 3101, PROF. J. ELDER 68
Definition of Theta 3n 2 + 7n + 8 = θ(n 2 ) c, c, n > 0: n n, c g( n) f( n) c g( n) 1 2 0 0 1 2 3 4 3 n 2 3n 2 + 7n + 8 4 n 2 COSC 3101, PROF. J. ELDER 69
Definition of Theta 3n 2 + 7n + 8 = θ(n 2 ) c, c, n > 0: n n, c g( n) f( n) c g( n) 1 2 0 0 1 2 COSC 3101, PROF. J. ELDER 70 1 3 1 2 3 1 2 + 7 1 + 8 4 1 2 False
Definition of Theta 3n 2 + 7n + 8 = θ(n 2 ) c, c, n > 0: n n, c g( n) f( n) c g( n) 1 2 0 0 1 2 COSC 3101, PROF. J. ELDER 71 7 3 7 2 3 7 2 + 7 7 + 8 4 7 2 False
Definition of Theta 3n 2 + 7n + 8 = θ(n 2 ) c, c, n > 0: n n, c g( n) f( n) c g( n) 1 2 0 0 1 2 COSC 3101, PROF. J. ELDER 72 8 3 8 2 3 8 2 + 7 8 + 8 4 8 2 True
Definition of Theta 3n 2 + 7n + 8 = θ(n 2 ) c, c, n > 0: n n, c g( n) f( n) c g( n) 1 2 0 0 1 2 COSC 3101, PROF. J. ELDER 73 9 3 9 2 3 9 2 + 7 9 + 8 4 9 2 True
Definition of Theta 3n 2 + 7n + 8 = θ(n 2 ) c, c, n > 0: n n, c g( n) f( n) c g( n) 1 2 0 0 1 2 10 COSC 3101, PROF. J. ELDER 74 3 10 2 3 10 2 + 7 10 + 8 4 10 2 True
Definition of Theta 3n 2 + 7n + 8 = θ(n 2 ) c, c, n > 0: n n, c g( n) f( n) c g( n) 1 2 0 0 1 2? 3 n 2 3n 2 + 7n + 8 4 n 2 COSC 3101, PROF. J. ELDER 75
Definition of Theta 3n 2 + 7n + 8 = θ(n 2 ) c, c, n > 0: n n, c g( n) f( n) c g( n) 1 2 0 0 1 2 8 n 8 3 n 2 3n 2 + 7n + 8 4 n 2 COSC 3101, PROF. J. ELDER 76
Definition of Theta 3n 2 + 7n + 8 = θ(n 2 ) c, c, n > 0: n n, c g( n) f( n) c g( n) 1 2 0 0 1 2 n 8 COSC 3101, PROF. J. ELDER 77 3 n 2 3n 2 + 7n + 8 4 n 2 True
Definition of Theta 3n 2 + 7n + 8 = θ(n 2 ) c, c, n > 0: n n, c g( n) f( n) c g( n) 1 2 0 0 1 2 n 8 COSC 3101, PROF. J. ELDER 78 3 n 2 3n 2 + 7n + 8 4 n 2 7n + 8 1 n 2 7 + 8 / n 1 n True
Definition of Theta 3n 2 + 7n + 8 = θ(n 2 ) True c, c, n > 0: n n, c g( n) f( n) c g( n) 1 2 0 0 1 2 3 4 8 n 8 3 n 2 3n 2 + 7n + 8 4 n 2 COSC 3101, PROF. J. ELDER 79
Asymptotic Notation Theta f(n) = θ(g(n)) f(n) c g(n) BigOh f(n) = O(g(n)) f(n) c g(n) Omega f(n) = Ω(g(n)) f(n) c g(n) Little Oh f(n) = o(g(n)) f(n) << c g(n) Little Omega f(n) = ω(g(n)) f(n) >> c g(n) COSC 3101, PROF. J. ELDER 80
Definition of Theta 3n 2 + 7n + 8 = θ(n) c, c, n > 0: n n, c g( n) f( n) c g( n) 1 2 0 0 1 2 COSC 3101, PROF. J. ELDER 81
Definition of Theta 3n 2 + 7n + 8 = θ(n) c, c, n > 0: n n, c g( n) f( n) c g( n) 1 2 0 0 1 2?? c 1 n 3n 2 + 7n + 8 c 2 n COSC 3101, PROF. J. ELDER 82
Definition of Theta 3n 2 + 7n + 8 = θ(n) c, c, n > 0: n n, c g( n) f( n) c g( n) 1 2 0 0 1 2 3 100 3 n 3n 2 + 7n + 8 100 n COSC 3101, PROF. J. ELDER 83
Definition of Theta 3n 2 + 7n + 8 = θ(n) c, c, n > 0: n n, c g( n) f( n) c g( n) 1 2 0 0 1 2? 3 n 3n 2 + 7n + 8 100 n COSC 3101, PROF. J. ELDER 84
Definition of Theta 3n 2 + 7n + 8 = θ(n) c, c, n > 0: n n, c g( n) f( n) c g( n) 1 2 0 0 1 2 100 COSC 3101, PROF. J. ELDER 85 3 100 3 100 2 + 7 100 + 8 100 100 False
Definition of Theta 3n 2 + 7n + 8 = θ(n) c, c, n > 0: n n, c g( n) f( n) c g( n) 1 2 0 0 1 2 3 10,000 3 n 3n 2 + 7n + 8 10,000 n COSC 3101, PROF. J. ELDER 86
Definition of Theta 3n 2 + 7n + 8 = θ(n) c, c, n > 0: n n, c g( n) f( n) c g( n) 1 2 0 0 1 2 10,000 COSC 3101, PROF. J. ELDER 87 3 10,000 3 10,000 2 + 7 10,000 + 8 10,000 10,000 False
Definition of Theta 3n 2 + 7n + 8 = θ(n) What is the reverse statement? c, c, n > 0: n n, c g( n) f( n) c g( n) 1 2 0 0 1 2 COSC 3101, PROF. J. ELDER 88
Understand Quantifiers!!! b, Loves(b, MM) [ b, Loves(b, MM)] [ b, Loves(b, MM)] b, Loves(b, MM) Sam Mary Sam Mary Bob Beth Bob Beth John Marilyn Monroe John Marilyn Monroe Fred Ann Fred Ann COSC 3101, PROF. J. ELDER 89
Definition of Theta 3n 2 + 7n + 8 = θ(n) The reverse statement c, c, n > 0: n n, c g( n) f( n) c g( n) 1 2 0 0 1 2 c, c, n > 0: n n : f( n) < c g( n) or f( n) > c g( n) 1 2 0 0 1 2 COSC 3101, PROF. J. ELDER 90
Definition of Theta 3n 2 + 7n + 8 = θ(n) c, c, n > 0: n n : f( n) < c g( n) or f( n) > c g( n) 1 2 0 0 1 2? 2 3 7 8 n + n + > c n 7 8 c 3 + + > Satisfied for n = max( n, c ) 2 2 n 2 n n 0 2 COSC 3101, PROF. J. ELDER 91
Order of Quantifiers f(n) = θ(g(n)) c, c, n > 0: n n, c g( n) f( n) c g( n) 1 2 0 0 1 2 n > 0: n n, c, c : c g( n) f( n) c g( n) 0 0 1 2 1 2? COSC 3101, PROF. J. ELDER 92
Understand Quantifiers!!! g, b, loves( b, g) b, g, loves( b, g) One girl Could be a separate girl for each boy. Sam Mary Sam Mary Bob Beth Bob Beth John Marilin Monro John Marilin Monro Fred Ann Fred Ann COSC 3101, PROF. J. ELDER 93
Order of Quantifiers f(n) = θ(g(n)) c, c, n > 0: n n, c g( n) f( n) c g( n) 1 2 0 0 1 2 n > 0: n n, c, c : c g( n) f( n) c g( n) 0 0 1 2 1 2 No! It cannot be a different c 1 and c 2 for each n. COSC 3101, PROF. J. ELDER 94
Other Notations Theta f(n) = θ(g(n)) f(n) c g(n) BigOh f(n) = O(g(n)) f(n) c g(n) Omega f(n) = Ω(g(n)) f(n) c g(n) Little Oh f(n) = o(g(n)) f(n) << c g(n) Little Omega f(n) = ω(g(n)) f(n) >> c g(n) COSC 3101, PROF. J. ELDER 95
Definition of Big Omega f( n) cg( n) f( n) Ω( g( n)) gn ( ) > 0:, ( ) ( ) cn, 0 n n0 f n cgn n COSC 3101, PROF. J. ELDER 96
Definition of Big Oh cg( n) f( n) f( n) O( g( n)) gn ( ) > 0:, ( ) ( ) cn, 0 n n0 f n cgn n COSC 3101, PROF. J. ELDER 97
Definition of Little Omega f( n) cg( n) f n ( ) ω( g( n)) gn ( ) c > 0, n > 0 : n n, f( n) > cg( n) 0 0 n COSC 3101, PROF. J. ELDER 98
Definition of Little Oh cg( n) f( n) f( n) o( g( n)) gn ( ) c > 0, n > 0 : n n, f( n) < cg( n) 0 0 n COSC 3101, PROF. J. ELDER 99
Time Complexity Is a Function Specifies how the running time depends on the size of the input. A function mapping size of input executed. time T(n) COSC 3101, PROF. J. ELDER 100
Definition of Time? COSC 3101, PROF. J. ELDER 101
Definition of Time # of seconds (machine dependent). # lines of code executed. # of times a specific operation is performed (e.g., addition). These are all reasonable definitions of time, because they are within a constant factor of each other. COSC 3101, PROF. J. ELDER 102
Definition of Input Size A good definition: #bits Examples: 1. Sorting an array A of k-bit e.g., k=16. Input size n = k length( A) integers. Note that, since n length(a), sufficient to define n = length( A). 2. Multiplying A,B. Let na = #bits used to represent A Let n = #bits used to represent B B Then input size n = na + nb Note that na log 2A and nb log2b T hus input size n log A + log B 2 2 COSC 3101, PROF. J. ELDER 103
Time Complexity Is a Function Specifies how the running time depends on the size of the input. A function mapping: size of input time T(n) executed. Or more precisely: # of bits n needed to represent the input # of operations T(n) executed. COSC 3101, PROF. J. ELDER 104
Time Complexity of an Algorithm The time complexity of an algorithm is the largest time required on any input of size n. (Worst case analysis.) O(n 2 ): For any input size n>n 0, the algorithm takes no more than cn 2 time on every input. Ω(n 2 ): For any input size n>n 0, the algorithm takes at least cn 2 time on at least one input. θ (n 2 ): Do both. COSC 3101, PROF. J. ELDER 105
End of Lecture 2 Monday, Mar 9, 2009
What is the height of tallest person in the class? Bigger than this? Smaller than this? Need to find only one person who is taller Need to look at every person COSC 3101, PROF. J. ELDER 107
Time Complexity of a Problem The time complexity of a problem is the time complexity of the fastest algorithm that solves the problem. O(n 2 ): Provide an algorithm that solves the problem in no more than this time. Remember: for every input, i.e. worst case analysis! Ω(n 2 ): Prove that no algorithm can solve it faster. Remember: only need one input that takes at least this long! θ (n 2 ): Do both. COSC 3101, PROF. J. ELDER 108