Computational Complexity. This lecture. Notes. Lecture 02 - Basic Complexity Analysis. Tom Kelsey & Susmit Sarkar. Notes

Transcription

1 Computational Complexity Lecture 02 - Basic Complexity Analysis Tom Kelsey & Susmit Sarkar School of Computer Science University of St Andrews twk@st-andrews.ac.uk Tom Kelsey and Susmit Sarkar CS3052-CC This lecture Time complexity analysis of actual algorithms Emphasise use of of O(n), Ω(n), and Θ(n) Time complexity is a combination of two factors 1 Time taken to do simple tasks on units of input 2 Frequency of those tasks The first of these is system dependent The second relates to the algorithm So abstract away from the first And concentrate on the frequencies For this lecture, a computer is what you think it is Tom Kelsey and Susmit Sarkar CS3052-CC Algorithm 1 generic for-loop code Require: Input X with X = n 1: Do c a things to initialise 2: for i = 1 to n do 3: c i things to X i 4: for j = 1 to n do 5: c j things to X j 6: c ij things to X i and X j 7: for k = 1 to n do 8: c k things to X k 9: c ik things to X i and X k 10: c jk things to X j and X k 11: c ijk things to X i, X j and X k 12: end for 13: end for 14: end for 15: Do c b things to return result Tom Kelsey and Susmit Sarkar CS3052-CC

2 Exact calculation of T(n) Constant time: (c a + c b )n 0 Linear time: (c i + c j + c k )n Quadratic time: (c ij + c jk + c ik )n 2 Cubic time: (c ijk )n 3 Large c means either "do many cheap things" or "do a few expensive things", or "do many expensive things" If c ijk is small, and (say) (c ij + c jk + c ik ) is large, then the quadratic term will dominate for small n After some n 0 the cubic term will dominate, for any positive c ijk Tom Kelsey and Susmit Sarkar CS3052-CC Exact calculation of T(n) Suppose (c a + c b ) = 1000 Suppose (c i + c j + c k ) = 100 Suppose (c ij + c jk + c ik ) = 3 Suppose (c ijk ) = 2 n has to be more than 10 before linear dominates constant effort, and more than 7 before n 3 dominates everything (since 8 3 = 512) T(n) 47n 4 n, so T(n) O(n 4 ) T(n) 0.01n 2 n, so T(n) Ω(n 2 ) If n 0 = 8 and c = c ijk, T(n) Ω(n 3 ) and O(n 3 ), hence T(n) Θ(n 3 ) Tom Kelsey and Susmit Sarkar CS3052-CC Asymptotic Order of Growth K. Wayne Princeton Tom Kelsey and Susmit Sarkar CS3052-CC

3 Asymptotic Order of Growth K. Wayne Princeton Tom Kelsey and Susmit Sarkar CS3052-CC Simplifying rules If some function g is an upper bound for your cost, then any upper bound for g is also an upper bound for your cost If some function g is a lower bound... If some function g is a tight bound... If some function h is an upper bound for two costs, then h is also an upper bound for the sum of those costs If some function h is a lower bound... If some function h is a tight bound... If some functions h and g are upper/lower/tight bounds for two costs, then the maximum of these is also an upper/lower/tight bound for the sum of those costs If some functions h and g are upper/lower/tight bounds for two costs, then the product of these is also an upper/lower/tight bound for the product of those costs Tom Kelsey and Susmit Sarkar CS3052-CC Asymptotic Order of Growth K. Wayne Princeton Tom Kelsey and Susmit Sarkar CS3052-CC

4 Asymptotic Order of Growth K. Wayne Princeton Tom Kelsey and Susmit Sarkar CS3052-CC Algorithm 2 example 1 Require: Input X with X = n 1: sum = 0 2: for i = 1 to n do 3: for j = 1 to n do 4: sum sum + 1 5: end for 6: end for 7: for k = 1 to n do 8: X k k 9: end for 10: return X Assignment is constant c 1 Final for-loop takes c 2 n = Θ(n) time Double loop involves each combination of (i, j) values i.e Θ(n 2 ) time Total time is Θ(c 1 + c 2 n + c 3 n 2 ) Using a simplifying rule, we only need the maximum of these functions So time is Θ(n 2 ) Tom Kelsey and Susmit Sarkar CS3052-CC Algorithm 3 example 2 1: sum1 = 0 2: for i = 1 to n do 3: for j = 1 to n do 4: sum1 sum : end for 6: end for 7: sum2 = 0 8: for i = 1 to n do 9: for j = 1 to i do 10: sum2 sum : end for 12: end for Assignments are constant c 1 First double loop takes c 2 n = Θ(n 2 ) time Second double loop takes c 3 ( n 1 + n) time i.e Θ(n 2 ) time Total time is Θ(c 1 + c 2 n 2 + ĉ 3 n 2 ) Using a simplifying rule, we only need the maximum of these functions So time is Θ(n 2 ), even though 2nd loop does half as much Tom Kelsey and Susmit Sarkar CS3052-CC

5 Algorithm 4 example 3 Require: n is a power of 2 1: sum1 = 0 2: for i = 1; i n; i = 2 do 3: for j = 1 to n do 4: sum1 sum : end for 6: end for 7: sum2 = 0 8: for i = 1; i n; i = 2 do 9: for j = 1 to i do 10: sum2 sum2 3 11: end for 12: end for Assignments are constant c 1 First double loop takes c 2 n log 2 n time i.e Θ(n log n) time Second double loop takes c 3 n time i.e Θ(n) time Total time is Θ(c 1 + c 2 n log n + c 3 n) Using a simplifying rule, we only need the maximum of these functions So time is Θ(n log n) Tom Kelsey and Susmit Sarkar CS3052-CC Problems with exact calculations Not always possible In fact, usually not possible In the examples, I used the two closed form solutions for summations given below These are not available, in general "Exact" only refers to frequencies, which are part of the algorithm Says nothing about assumed constants costs which are part of the system being used As we ll see, use of e.g. fast memory can drastically alter performance log(n) n = n log n, i=1 log(n) 2 i = 2 log(n) 1 = n 1 i=1 Tom Kelsey and Susmit Sarkar CS3052-CC More problems with exact calculations Neither exponents nor exponands have to be integral Strassen s algorithm is O(n ) 3-SAT using Speckenmeyer & Moniens 1985 method is O( n ) Usually straightforward to derive an upper bound So we often see big-o used as "the tightest bound we have derived so far" Often good theoretical lower bounds can be found, so we know that the exact worst-case time complexity is between the two Max Independent Set using a method due to Fomin et al. is O( n ) Lower bound of Ω( n ) for the (unknown) worst case running time Tom Kelsey and Susmit Sarkar CS3052-CC

6 What about average- and best case analysis? All our contrived examples take an exact number of steps for fixed n Most algorithms have if-statements if some conditions hold, stop, skip, or do less work roughly speaking, this is how we optimise our code Worst-case analysis assumes that none of these shortcuts happen we usually have to devise pathological inputs Best-case typically assumes that all shortcuts happen we devise optimal inputs Average-case involves empirical evaluation in principle this is simple run the code loads of times on loads of inputs, find a function that best describes the data but now we can t avoid system issues such as compiler flags, L2 cache use, pointer tricks, etc. Tom Kelsey and Susmit Sarkar CS3052-CC Empirical problems In Computer Science, the best framework for empirical evaluation a computer system is far from ideal I can run the same code with the same inputs on the same system and get wildly different results garbage collector kicks in, cron job starts, etc. We find it hard to control for all the possibly-confounding variables Many published CS methods can not be replicated Serious CS research communities do their best to overcome these problems libraries of test problems, competitions at conferences, evaluate old vs new on the same system, etc. CS is not as empirically rigorous as Physics, Chemistry, Medicine or Biology sometimes due to unscientific practice by people who should know better often for very good reasons Tom Kelsey and Susmit Sarkar CS3052-CC Divide & conquer The last few slides motivate why courses on this topic usually start with abstract models of computation I want to make sure that you are comfortable the terminology and results I use in those lectures So we re continuing with analysis of actual algorithms Recursive methods are used extensively some languages only allow recursive functions The analysis for for- and while-loops doesn t work Many recursive methods involve chopping the input into manageable chunks, then doing something to the chunks Instead of attacking the input as a monolithic entity This algorithmic design pattern is known as divide & conquer And we can often deduce complexity results from the associated recurrence relations Tom Kelsey and Susmit Sarkar CS3052-CC

7 Algorithm 5 Mergesort 1: procedure MSORT(X,L,R) 2: mid (L + R)/2 3: if L < R then 4: MSORT(X,L,mid) 5: MSORT(X,mid + 1,R) 6: COMBINE(X,L,mid,R) 7: end if 8: end procedure X consists of n comparables Strictly speaking, getting R is Θ(n) itself Let T(n) be time needed by MSORT Clearly T(1) = O(1) T(n) = 2T(n/2) + O(n) Two recursive calls Each applied to half the input the O(n) is for combine write O(n) as n to simplify things Tom Kelsey and Susmit Sarkar CS3052-CC Mergesort solve the RR by hand T(n) = 2T(n/2) + n = 2[2T(n/4) + n/2] + n = 4T(n/4) + 2n = 4[2T(n/8) + n/4] + 2n = 8T(n/8) + 3n =... = 16T(n/16) + 4n =... = 2 k T(n/2 k ) + kn = 2 log2n T(1) + (log 2 (n))n (since log 2 (n) = k) = n + (log 2 (n))n = O(n log 2 (n)) Tom Kelsey and Susmit Sarkar CS3052-CC Problems with solving by analysis That was a lot of effort for a simple result, and it might not generalise to other procedures We needed to make a simplifying assumption, and we still lack an induction proof that the solution of the RR is of the form n log(n) Deriving the starting RR was simple, though Count the recursive calls, spot the rate of division of the input, and calculate the work needed at each conquer stage We d like to be able to solve these RRs by inspection of a case split most of the time It turns out that this can be done Tom Kelsey and Susmit Sarkar CS3052-CC

8 Tom Kelsey and Susmit Sarkar CS3052-CC Mergesort using the MT T(n) = 2T(n/2) + n, so a = b = 2 and f (n) = n Case 1: n = O(n log2(2) ɛ )? = O(n 1 ɛ )? Case 2: n = Θ(n log k (n)) for some k 0? = Θ(n)? k = 0 works, so T(n) = Θ(n log(n)) Tom Kelsey and Susmit Sarkar CS3052-CC Algorithm 6 Example: binary search 1: procedure BINSCH(X,lo,hi,x) 2: mid lo + (hi lo)/2 3: if x < X mid then 4: return BINSCH(X,lo,mid 1,x) 5: else if x > X mid then 6: return BINSCH(X,mid + 1,hi,x) 7: else 8: return mid 9: end if 10: end procedure X consists of n ordered comparables As before, we ignore cost of getting input Let T(n) be time needed by BINSCH T(1) isn t needed T(n) = T(n/2) + O(1) At most one recursive call at any level Each applied to half the input the O(1) is for comparisons and returns Tom Kelsey and Susmit Sarkar CS3052-CC

9 Binary Search using the MT T(n) = T(n/2) + 1, so a = 1, b = 2 and f (n) = K Case 1: Case 2: K = O(n log 2(1) ɛ )? = O(n 0 ɛ )? K = Θ(n log 2(1) log k (n)) for some k 0? = Θ(n 0 log k (n)) for some k 0? = Θ(1 log 0 (n))? = Θ(1)? Tom Kelsey and Susmit Sarkar CS3052-CC Tom Kelsey and Susmit Sarkar CS3052-CC Tree Traversal & Print Paths using the MT Some combination of recurse left, print and recurse right check your notes for CS2001 last year T(n) = 2T(n/2) + O(1), so a = 2, b = 2 and f (n) = K Case 1: K = O(n log 2(2) ɛ ) for some ɛ 0? = O(n 1 ɛ )? Any ɛ in (0, 1) works, so T(n) = Θ(n log 2(2) ) = Θ(n) Tom Kelsey and Susmit Sarkar CS3052-CC

10 Master Theorem examples T(n) = 4T(n/2) + log(n), so a = 4, b = 2 and f (n) = log(n) Case 1: log(n) = O(n log2(4) ɛ ) for some ɛ > 0? = O(n 2 ɛ )? Any ɛ in (0, 1) works, so T(n) = Θ(n log 2(4) ) = Θ(n 2 ) Tom Kelsey and Susmit Sarkar CS3052-CC Master Theorem examples T(n) = 2T(n/2) + n log(n), so a = 2, b = 2 and f (n) = n log(n) Case 2: n log(n) = Θ(n log 2(2) log k (n)) for some k 0? = Θ(n 1 log k (n)) for some k 0? = Θ(n log 1 (n))? = Θ(n log(n)) k = 1 works, so T(n) = Θ(n log 2 (n)) Tom Kelsey and Susmit Sarkar CS3052-CC Master Theorem examples T(n) = 6T(n/3) + n 2 log(n), so a = 6, b = 3 and f (n) = n 2 log(n) Case 3: n 2 log(n) = Ω(n log 3(6)+ɛ ) for some ɛ > 0? log 3 (6) 1.6, so any ɛ < 2 log 3 (6) will do Still have to check the regularity condition: 6 n2 ( n ) 3 log cn 2 log(n)? 3 for positive c and large enough n. Any c bigger than 2 will do, so the result holds Tom Kelsey and Susmit Sarkar CS3052-CC

11 Master Theorem Summary We have a divide-and-conquer algorithm with known rate of division, number of recursive calls, and amount of work done at each level of recursion If the amount of work is larger than total recursive work, then this will dominate for f (n) = 2 n or f (n) = n! case 3 almost always holds If the amount of work is negligible at each recursion, then the recursive calls dominate case 1 It often happens that the amount of work adds log terms to recursive complexity case 2 So we can use the Master Theorem as a shortcut, avoiding the "correct" way to solve such RRs Tom Kelsey and Susmit Sarkar CS3052-CC Master Theorem Limitations Not applicable to many recursive methods Sequential search: T(n) = T(n 1) + O(1) Selection sort: T(n) = T(n 1) + O(n) Naïve Fibonacci: F(n) = F(n 1) + F(n 2) Obvious source of tricky exam questions for undergraduates For some problems, the shortcut may be no easier than the correct method: Guess an upper bound for the complexity Produce a proof by induction that validates your guess Guess a lower bound... Prove by induction... If both guesses were the same great, otherwise start tightening from above and/or below Tom Kelsey and Susmit Sarkar CS3052-CC Master Theorem Avoidance Most divide & conquer methods have RRs of the form T(n) = at(n/b) + f (n) Imagine a recursion tree with branching rate a and node values f (n/b i ) at depth i. Assume that T(1) = 1 (since we only care about asymptotic bounds) and make couple of other assumptions that do hold in general Now T(n) is the sum of all the nodes in the tree, and we can use a computer algebra system Maple or Mathematica to search for a closed form solution, or some simplified form T(n) = L i=1 a i f (n/b i ) Tom Kelsey and Susmit Sarkar CS3052-CC

12 Master Theorem Avoidance Even if there is no closed form solution, the series are often geometric the final term is larger than all its predecessors In which case, the final term is the dominating function we use for the complexity For mergesort we have 2 i nodes at each of log(n) levels, each with value n/2 i : log(n) T(n) = n i=1 Using the closed form from an earlier slide, T(n) = Θ(n log(n)) Tom Kelsey and Susmit Sarkar CS3052-CC Tom Kelsey and Susmit Sarkar CS3052-CC Naïve Fibonacci Possibly mentioned in CS2001 last year Without memoisation, we recompute each sub-value on every recursive branch that it appears Which is either all of them or nearly all of them So we conjecture that F(n) O(r n ) for some r > 1 F(n) = F(n 1) + F(n 2) αr n 1 + αr n 2 our guess αr n? So we want r 2 r 1 0, which is fine since the largest root is the golden ratio φ Now use the base cases F(0) φ 0 = 0 F(1) = 0 and 1 φ 1 = 1 φ > 0 Tom Kelsey and Susmit Sarkar CS3052-CC

13 Naïve Fibonacci We now have the two numbers needed to show that F(n) is O(r n ) φ is the exponand greater than 1, r 1/φ is the positive constant α with F(n) αr n n Trying the same with lower bounds F(n) βr n is not easy But we don t need a lower bound for all n, just above a certain n 0 Setting N 0 as 2 makes life much easier Just one base case F(2) φ 2 = 1 φ < 1 φ Tom Kelsey and Susmit Sarkar CS3052-CC Naïve Fibonacci We now have the three numbers needed to show that F(n) is Ω(r n ) φ is the exponand greater than 1, r 1/φ 2 is the positive constant β which is less than α n 0 = 2 with F(n) βr n n n 0 We have upper and lower bounds with the same exponand So we have a tight bound F(n) = Θ(φ n ) n 2 This is more Maths than CS, but a lot of complexity analysis is inherently mathematical We ll do a couple of more straightforward examples Tom Kelsey and Susmit Sarkar CS3052-CC Towers of Hanoi 1 Move the top n 1 disks from Tower 1 to Tower 2 2 Move the nth disk from Tower 1 to Tower 3 3 Move the n 1 disks from Tower 2 to Tower 3 Calculate for small n: T(n) = 2T(n 1) + 1, T(0) = 0 T(0) = 0, T(1) = 1, T(2) = 3, T(3) = 7, T(4) = 15, T(5) = 31, T(6) = 63,... For this course only I ll accept the above as justification for the assertion that 1 T(n) = 2 n 1 2 the algorithm is Θ(2 n ) Tom Kelsey and Susmit Sarkar CS3052-CC

14 Naïve Fibonacci again The reason I used r ealier is because it s a well-known exact example In general, we wouldn t expect things to work out so nicely So we conjecture that F(n) O(2 n ) F(n) = F(n 1) + F(n 2) α2 n 1 + α2 n 2 our guess α2 n? So we want 2 n 2 n n 2, which is true for n 2 We can forget about α and any base cases, and assert that F(n) = O(2 n ) And let someone else worry about lower and tight bounds Tom Kelsey and Susmit Sarkar CS3052-CC Summary We can do exact time complexity analysis in some cases Simple for- and while-loop methods using closed form solutions of finite summations Divide & conquer recursive methods using patterns, the Master Theorem, or recursion trees or full induction proofs for each type of bound Calculating the exact complexity of complicated algorithms is at best not easy So we often use the best lower bound available Tom Kelsey and Susmit Sarkar CS3052-CC Distler & Kelsey, 2008 Tom Kelsey and Susmit Sarkar CS3052-CC

15 Next lecture We ve assumed properties of computational frameworks and algorithms And classes of problems We can justify the use of of O(n), Ω(n), and Θ(n) Essentially, they deal with our abstraction away from real systems and mitigate our inability to compute most exact results With this out of the way, we go back to a much higher level of abstraction No assumptions about computational frameworks, algorithms, or classes of problems What progress can we make at a fully abstracted level? Tom Kelsey and Susmit Sarkar CS3052-CC