Part II. Foundations. Ernst Mayr, Harald Räcke 16

Transcription

1 Part II Foundations Ernst Mayr, Harald Räcke 16

2 Vocabularies a b a times b a multiplied by b a into b a b a divided by b a by b a over b a b (a: numerator (Zähler), b: denominator (Nenner)) a raised to the b-th power a to the b-th a raised to the power of b a to the power of b a raised to b a to the b a raised by the exponent of b Ernst Mayr, Harald Räcke 17

3 Vocabularies n! n factorial ( n k) n choose k x i x subscript i x sub i x i log b a log to the base b of a log a to the base b f : X Y, x x 2 f is a function that maps from domain (Definitionsbereich) X to codomain (Zielmenge) Y The set {y Y x X : f (x) = y} is the image or the range of the function (Bildbereich/Wertebereich) Ernst Mayr, Harald Räcke 18

4 3 Goals Gain knowledge about efficient algorithms for important problems, ie, learn how to solve certain types of problems efficiently Learn how to analyze and judge the efficiency of algorithms Learn how to design efficient algorithms 3 Goals Ernst Mayr, Harald Räcke 19

5 4 Modelling Issues What do you measure? Memory requirement Running time Number of comparisons Number of multiplications Number of hard-disc accesses Program size Power consumption 4 Modelling Issues Ernst Mayr, Harald Räcke 20

6 4 Modelling Issues How do you measure? Implementing and testing on representative inputs How do you choose your inputs? May be very time-consuming Very reliable results if done correctly Results only hold for a specific machine and for a specific set of inputs Theoretical analysis in a specific model of computation Gives asymptotic bounds like this algorithm always runs in time O(n 2 ) Typically focuses on the worst case Can give lower bounds like any comparison-based sorting algorithm needs at least Ω(n log n) comparisons in the worst case 4 Modelling Issues Ernst Mayr, Harald Räcke 21

7 4 Modelling Issues Input length The theoretical bounds are usually given by a function f : N N that maps the input length to the running time (or storage space, comparisons, multiplications, program size etc) The input length may eg be the size of the input (number of bits) the number of arguments Example 1 Suppose n numbers from the interval {1,, N} have to be sorted In this case we usually say that the input length is n instead of eg n log N, which would be the number of bits required to encode the input 4 Modelling Issues Ernst Mayr, Harald Räcke 22

8 Model of Computation How to measure performance 1 Calculate running time and storage space etc on a simplified, idealized model of computation, eg Random Access Machine (RAM), Turing Machine (TM), 2 Calculate number of certain basic operations: comparisons, multiplications, harddisc accesses, Version 2 is often easier, but focusing on one type of operation makes it more difficult to obtain meaningful results 4 Modelling Issues Ernst Mayr, Harald Räcke 23

9 Turing Machine Very simple model of computation Only the current memory location can be altered Very good model for discussing computabiliy, or polynomial vs exponential time Some simple problems like recognizing whether input is of the form xx, where x is a string, have quadratic lower bound Not a good model for developing efficient algorithms control unit state state holds program and can act as constant size memory 4 Modelling Issues Ernst Mayr, Harald Räcke 24

10 Random Access Machine (RAM) Input tape and output tape (sequences of zeros and ones; unbounded length) Memory unit: infinite but countable number of registers R[0], R[1], R[2], Registers hold integers Indirect addressing input tape memory R[0] R[1] control unit R[2] R[3] Note that in the picture on the right the tapes are one-directional, and that a READ- or WRITE-operation always advances its tape output tape R[4] R[5] 4 Modelling Issues Ernst Mayr, Harald Räcke 25

11 Random Access Machine (RAM) Operations input operations (input tape R[i]) READ i output operations (R[i] output tape) WRITE i register-register transfers R[j] := R[i] R[j] := 4 indirect addressing R[j] := R[R[i]] loads the content of the R[i]-th register into the j-th register R[R[i]] := R[j] loads the content of the j-th into the R[i]-th register 4 Modelling Issues Ernst Mayr, Harald Räcke 26

12 Random Access Machine (RAM) Operations branching (including loops) based on comparisons jump x jumps to position x in the program; sets instruction counter to x; reads the next operation to perform from register R[x] jumpz x R[i] jump to x if R[i] = 0 if not the instruction counter is increased by 1; jumpi i jump to R[i] (indirect jump); arithmetic instructions: +,,, / R[i] := R[j] + R[k]; R[i] := -R[k]; The jump-directives are very close to the jump-instructions contained in the assembler language of real machines 4 Modelling Issues Ernst Mayr, Harald Räcke 27

13 Model of Computation uniform cost model Every operation takes time 1 logarithmic cost model The cost depends on the content of memory cells: The time for a step is equal to the largest operand involved; The storage space of a register is equal to the length (in bits) of the largest value ever stored in it Bounded word RAM model: cost is uniform but the largest value stored in a register may not exceed 2 w, where usually w = log 2 n The latter model is quite realistic as the word-size of a standard computer that handles a problem of size n must be at least log 2 n as otherwise the computer could either not store the problem instance or not address all its memory 4 Modelling Issues Ernst Mayr, Harald Räcke 28

14 4 Modelling Issues Example 2 Algorithm 1 RepeatedSquaring(n) 1: r 2; 2: for i = 1 n do 3: r r 2 4: return r running time: uniform model: n steps logarithmic model: n = 2 n+1 1 = Θ(2 n ) space requirement: uniform model: O(1) logarithmic model: O(2 n ) 4 Modelling Issues Ernst Mayr, Harald Räcke 29

15 There are different types of complexity bounds: best-case complexity: C bc (n) := min{c(x) x = n} Usually easy to analyze, but not very meaningful worst-case complexity: C wc (n) := max{c(x) x = n} Usually moderately easy to analyze; sometimes too pessimistic average case complexity: C avg (n) := 1 I n x =n more general: probability measure µ C avg (n) := C(x) x I n µ(x) C(x) cost of instance C(x) x x input length of instance x I n set of instances of length n 4 Modelling Issues Ernst Mayr, Harald Räcke 30

16 There are different types of complexity bounds: amortized complexity: The average cost of data structure operations over a worst case sequence of operations randomized complexity: The algorithm may use random bits Expected running time (over all possible choices of random bits) for a fixed input x Then take the worst-case over all x with x = n 4 Modelling Issues Ernst Mayr, Harald Räcke 30

17 4 Modelling Issues Bibliography [MS08] [CLRS90] Kurt Mehlhorn, Peter Sanders: Algorithms and Data Structures The Basic Toolbox, Springer, 2008 Thomas H Cormen, Charles E Leiserson, Ron L Rivest, Clifford Stein: Introduction to algorithms (3rd ed), McGraw-Hill, 2009 Chapter 21 and 22 of [MS08] and Chapter 2 of [CLRS90] are relevant for this section 4 Modelling Issues Ernst Mayr, Harald Räcke 31

18 5 Asymptotic Notation We are usually not interested in exact running times, but only in an asymptotic classification of the running time, that ignores constant factors and constant additive offsets We are usually interested in the running times for large values of n Then constant additive terms do not play an important role An exact analysis (eg exactly counting the number of operations in a RAM) may be hard, but wouldn t lead to more precise results as the computational model is already quite a distance from reality A linear speed-up (ie, by a constant factor) is always possible by eg implementing the algorithm on a faster machine Running time should be expressed by simple functions 5 Asymptotic Notation Ernst Mayr, Harald Räcke 31

19 Asymptotic Notation Formal Definition Let f denote functions from N to R + O(f ) = {g c > 0 n 0 N 0 n n 0 : [g(n) c f (n)]} (set of functions that asymptotically grow not faster than f ) Ω(f ) = {g c > 0 n 0 N 0 n n 0 : [g(n) c f (n)]} (set of functions that asymptotically grow not slower than f ) Θ(f ) = Ω(f ) O(f ) (functions that asymptotically have the same growth as f ) o(f ) = {g c > 0 n 0 N 0 n n 0 : [g(n) c f (n)]} (set of functions that asymptotically grow slower than f ) ω(f ) = {g c > 0 n 0 N 0 n n 0 : [g(n) c f (n)]} (set of functions that asymptotically grow faster than f ) 5 Asymptotic Notation Ernst Mayr, Harald Räcke 32

20 Asymptotic Notation There is an equivalent definition using limes notation (assuming that the respective limes exists) f and g are functions from N 0 to R + 0 g O(f ): g Ω(f ): g Θ(f ): g o(f ): g ω(f ): 0 lim n g(n) f (n) < 0 < lim n g(n) f (n) 0 < lim n g(n) f (n) < g(n) lim n f (n) = 0 g(n) lim n f (n) = Note that for the version of the Landau notation defined here, we assume that f and g are positive functions There also exist versions for arbitrary functions, and for the case that the limes is not infinity 5 Asymptotic Notation Ernst Mayr, Harald Räcke 33

21 Asymptotic Notation Abuse of notation 1 People write f = O(g), when they mean f O(g) This is not an equality (how could a function be equal to a set of functions) 2 People write f (n) = O(g(n)), when they mean f O(g), with f : N R +, n f (n), and g : N R +, n g(n) 3 People write eg h(n) = f (n) + o(g(n)) when they mean that there exists a function z : N R +, n z(n), z o(g) such that h(n) = f (n) + z(n) 2 In this context f (n) does not mean the function f evaluated at n, but instead it is a shorthand for the function itself (leaving out domain and codomain and only giving the rule of correspondence of the function) 3 This is particularly useful if you do not want to ignore constant factors For example the median of n elements can be determined using 3 2 n+o(n) comparisons 5 Asymptotic Notation Ernst Mayr, Harald Räcke 34

22 Asymptotic Notation Abuse of notation 4 People write O(f (n)) = O(g(n)), when they mean O(f (n)) O(g(n)) Again this is not an equality 5 Asymptotic Notation Ernst Mayr, Harald Räcke 34

23 Asymptotic Notation in Equations How do we interpret an expression like: 2n 2 + 3n + 1 = 2n 2 + Θ(n) Here, Θ(n) stands for an anonymous function in the set Θ(n) that makes the expression true Note that Θ(n) is on the right hand side, otw this interpretation is wrong 5 Asymptotic Notation Ernst Mayr, Harald Räcke 35

24 Asymptotic Notation in Equations How do we interpret an expression like: 2n 2 + O(n) = Θ(n 2 ) Regardless of how we choose the anonymous function f (n) O(n) there is an anonymous function g(n) Θ(n 2 ) that makes the expression true 5 Asymptotic Notation Ernst Mayr, Harald Räcke 36

25 Asymptotic Notation in Equations How do we interpret an expression like: The Θ(i)-symbol on the left represents one anonymous function f : N R +, and then i f (i) is computed n Θ(i) = Θ(n 2 ) i=1 Careful! It is understood that every occurence of an O-symbol (or Θ, Ω, o, ω) on the left represents one anonymous function Hence, the left side is not equal to Θ(1) + Θ(2) + + Θ(n 1) + Θ(n) Θ(1)+Θ(2)+ +Θ(n 1)+Θ(n) does not really have a reasonable interpretation 5 Asymptotic Notation Ernst Mayr, Harald Räcke 37

26 Asymptotic Notation in Equations We can view an expression containing asymptotic notation as generating a set: n 2 O(n) + O(log n) represents { f : N R + f (n) = n 2 g(n) + h(n) } with g(n) O(n) and h(n) O(log n) Recall that according to the previous slide eg the expressions n i=1 O(i) and n/2 i=1 O(i) + n i=n/2+1 O(i) generate different sets 5 Asymptotic Notation Ernst Mayr, Harald Räcke 38

27 Asymptotic Notation in Equations Then an asymptotic equation can be interpreted as containement btw two sets: n 2 O(n) + O(log n) = Θ(n 2 ) represents n 2 O(n) + O(log n) Θ(n 2 ) Note that the equation does not hold 5 Asymptotic Notation Ernst Mayr, Harald Räcke 39

28 Asymptotic Notation Lemma 3 Let f, g be functions with the property n 0 > 0 n n 0 : f (n) > 0 (the same for g) Then c f (n) Θ(f (n)) for any constant c O(f (n)) + O(g(n)) = O(f (n) + g(n)) O(f (n)) O(g(n)) = O(f (n) g(n)) O(f (n)) + O(g(n)) = O(max{f (n), g(n)}) The expressions also hold for Ω Note that this means that f (n) + g(n) Θ(max{f (n), g(n)}) 5 Asymptotic Notation Ernst Mayr, Harald Räcke 40

29 Asymptotic Notation Comments Do not use asymptotic notation within induction proofs For any constants a, b we have log a n = Θ(log b n) Therefore, we will usually ignore the base of a logarithm within asymptotic notation In general log n = log 2 n, ie, we use 2 as the default base for the logarithm 5 Asymptotic Notation Ernst Mayr, Harald Räcke 41

30 Asymptotic Notation In general asymptotic classification of running times is a good measure for comparing algorithms: If the running time analysis is tight and actually occurs in practise (ie, the asymptotic bound is not a purely theoretical worst-case bound), then the algorithm that has better asymptotic running time will always outperform a weaker algorithm for large enough values of n However, suppose that I have two algorithms: Algorithm A Running time f (n) = 1000 log n = O(log n) Algorithm B Running time g(n) = log 2 n Clearly f = o(g) However, as long as log n 1000 Algorithm B will be more efficient 5 Asymptotic Notation Ernst Mayr, Harald Räcke 42

31 5 Asymptotic Notation Bibliography [MS08] [CLRS90] Kurt Mehlhorn, Peter Sanders: Algorithms and Data Structures The Basic Toolbox, Springer, 2008 Thomas H Cormen, Charles E Leiserson, Ron L Rivest, Clifford Stein: Introduction to algorithms (3rd ed), McGraw-Hill, 2009 Mainly Chapter 3 of [CLRS90] [MS08] covers this topic in chapter 21 but not very detailed 5 Asymptotic Notation Ernst Mayr, Harald Räcke 43

32 6 Recurrences Algorithm 2 mergesort(list L) 1: n size(l) 2: if n 1 return L 3: L 1 L[1 n 2 ] 4: L 2 L[ n n] 5: mergesort(l 1 ) 6: mergesort(l 2 ) 7: L merge(l 1, L 2 ) 8: return L This algorithm requires ( n ) ( n ) ( n ) T (n) = T + T + O(n) 2T + O(n) comparisons when n > 1 and 0 comparisons when n 1 6 Recurrences Ernst Mayr, Harald Räcke 43

33 Recurrences How do we bring the expression for the number of comparisons ( running time) into a closed form? For this we need to solve the recurrence 6 Recurrences Ernst Mayr, Harald Räcke 44

34 Methods for Solving Recurrences 1 Guessing+Induction Guess the right solution and prove that it is correct via induction It needs experience to make the right guess 2 Master Theorem For a lot of recurrences that appear in the analysis of algorithms this theorem can be used to obtain tight asymptotic bounds It does not provide exact solutions 3 Characteristic Polynomial Linear homogenous recurrences can be solved via this method 6 Recurrences Ernst Mayr, Harald Räcke 45

35 Methods for Solving Recurrences 4 Generating Functions A more general technique that allows to solve certain types of linear inhomogenous relations and also sometimes non-linear recurrence relations 5 Transformation of the Recurrence Sometimes one can transform the given recurrence relations so that it eg becomes linear and can therefore be solved with one of the other techniques 6 Recurrences Ernst Mayr, Harald Räcke 46

36 61 Guessing+Induction First we need to get rid of the O-notation in our recurrence: { ( n ) 2T 2 + cn n 2 T (n) 0 otherwise Assume that instead we had { ( n ) 2T 2 + cn n 2 T (n) 0 otherwise One way of solving such a recurrence is to guess a solution, and check that it is correct by plugging it in 61 Guessing+Induction Ernst Mayr, Harald Räcke 47

37 61 Guessing+Induction Suppose we guess T (n) dn log n for a constant d Then if we choose d c ( n ) T (n) 2T + cn 2 ( 2 d n 2 log n ) + cn 2 = dn(log n 1) + cn = dn log n + (c d)n dn log n Formally one would make an induction proof, where the above is the induction step The base case is usually trivial 61 Guessing+Induction Ernst Mayr, Harald Räcke 48

38 61 Guessing+Induction Guess: T (n) dn log n Proof (by induction) { ( n ) 2T 2 + cn n 16 T (n) b otw base case (2 n < 16): true if we choose d b induction step 2 n 1 n: Suppose statem is true for n {2,, n 1}, and n 16 We prove it for n: ( n ) T (n) 2T + cn 2 ( 2 d n 2 log n ) + cn 2 = dn(log n 1) + cn = dn log n + (c d)n dn log n Hence, statement is true if we choose d c Note that this proves the statement for n N 2, as the statement is wrong for n = 1 The base case is usually omitted, as it is the same for different recurrences

39 61 Guessing+Induction Why did we change the recurrence by getting rid of the ceiling? If we do not do this we instead consider the following recurrence: { 2T ( n 2 ) + cn n 16 T (n) b otherwise Note that we can do this as for constant-sized inputs the running time is always some constant (b in the above case) 61 Guessing+Induction Ernst Mayr, Harald Räcke 50

40 61 Guessing+Induction We also make a guess of T (n) dn log n and get n 2 n n n ( n ) T (n) 2T 2 ( n 2 d 2 + cn log n 2 ) + cn 2 ( d(n/2 + 1) log(n/2 + 1) ) + cn ( 9 ) dn log 16 n + 2d log n + cn log 9 16 n = log n + (log 9 4) log n n 4 = dn log n + (log 9 4)dn + 2d log n + cn dn log n + (log 9 35)dn + cn dn log n 033dn + cn dn log n for a suitable choice of d 61 Guessing+Induction Ernst Mayr, Harald Räcke 51

41 62 Master Theorem Note that the cases do not cover all possibilities Lemma 4 Let a 1, b 1 and ɛ > 0 denote constants Consider the recurrence ( n ) T (n) = at + f (n) b Case 1 If f (n) = O(n log b(a) ɛ ) then T (n) = Θ(n log b a ) Case 2 If f (n) = Θ(n log b(a) log k n) then T (n) = Θ(n log b a log k+1 n), k 0 Case 3 If f (n) = Ω(n logb(a)+ɛ ) and for sufficiently large n af ( n b ) cf (n) for some constant c < 1 then T (n) = Θ(f (n)) 62 Master Theorem Ernst Mayr, Harald Räcke 52

42 62 Master Theorem We prove the Master Theorem for the case that n is of the form b l, and we assume that the non-recursive case occurs for problem size 1 and incurs cost 1 62 Master Theorem Ernst Mayr, Harald Räcke 53

43 The Recursion Tree The running time of a recursive algorithm can be visualized by a recursion tree: nx a f (n) n b a n b a n b a af ( n b ) n b 2 n b 2 n b 2 n b 2 n b 2 a a a a a a a a a n b 2 n b 2 n b 2 n b 2 a 2 f ( n b 2 ) a log b n = n log b a 62 Master Theorem Ernst Mayr, Harald Räcke 54

44 62 Master Theorem This gives log b n 1 ( ) T (n) = n log b n a + a i f b i i=0 62 Master Theorem Ernst Mayr, Harald Räcke 55

45 Case 1 Now suppose that f (n) cn log b a ɛ T (n) n log b a = b i(log b a ɛ) = b ɛi (b log b a ) i = b ɛi a i Hence, k i=0 qi = qk+1 1 q 1 ( ) c T (n) b ɛ log b n 1 i=0 ( ) n a i f b i log b n 1 ( ) n logb c a i a ɛ b i i=0 = cn log b a ɛ log b n 1 i=0 ( b ɛ ) i = cn log b a ɛ (b ɛ log b n 1)/(b ɛ 1) = cn log b a ɛ (n ɛ 1)/(b ɛ 1) c = b ɛ 1 nlog b a (n ɛ 1)/(n ɛ ) n log b(a) T (n) = O(n log b a ) 62 Master Theorem Ernst Mayr, Harald Räcke 56

46 Case 2 Now suppose that f (n) cn log b a T (n) n log b a = log b n 1 i=0 ( ) n a i f b i log b n 1 ( ) n logb c a i a b i i=0 log b n 1 = cn log b a 1 i=0 = cn log b a log b n Hence, T (n) = O(n log b a log b n) T (n) = O(n log b a log n) 62 Master Theorem Ernst Mayr, Harald Räcke 57

47 Case 2 Now suppose that f (n) cn log b a T (n) n log b a = log b n 1 i=0 ( ) n a i f b i log b n 1 ( ) n logb c a i a b i i=0 log b n 1 = cn log b a 1 i=0 = cn log b a log b n Hence, T (n) = Ω(n log b a log b n) T (n) = Ω(n log b a log n) 62 Master Theorem Ernst Mayr, Harald Räcke 58

48 Case 2 Now suppose that f (n) cn log b a (log b (n)) k T (n) n log b a = n = b l l = log b n log b n 1 i=0 ( ) n a i f b i log b n 1 ( ) n logb c a i a b i i=0 = cn log b a l 1 i=0 ( ( b l )) k log b l 1 = cn log b a (l i) k = cn log b a i=0 l i=1 i k c k nlog b a l k+1 b i 1 k lk+1 ( ( )) n k log b b i T (n) = O(n log b a log k+1 n) 62 Master Theorem Ernst Mayr, Harald Räcke 59

49 Case 3 Now suppose that f (n) dn log b a+ɛ, and that for sufficiently large n: af (n/b) cf (n), for c < 1 From this we get a i f (n/b i ) c i f (n), where we assume that n/b i 1 n 0 is still sufficiently large T (n) n log b a = q < 1 : n i=0 qi = 1 qn+1 1 q 1 1 q Hence, log b n 1 i=0 log b n 1 i=0 ( ) n a i f b i c i f (n) + O(n log b a ) 1 1 c f (n) + O(nlog b a ) T (n) O(f (n)) T (n) = Θ(f (n)) Where did we use f (n) Ω(n log b a+ɛ )? 62 Master Theorem Ernst Mayr, Harald Räcke 60

50 Example: Multiplying Two Integers Suppose we want to multiply two n-bit Integers, but our registers can only perform operations on integers of constant size For this we first need to be able to add two integers A and B: A B This gives that two n-bit integers can be added in time O(n) 62 Master Theorem Ernst Mayr, Harald Räcke 61

51 Example: Multiplying Two Integers Suppose that we want to multiply an n-bit integer A and an m-bit integer B (m n) This is also nown as the school method for multiplying integers Note that the intermediate numbers that are generated can have at most m + n 2n bits Time requirement: Computing intermediate results: O(nm) Adding m numbers of length 2n: O((m + n)m) = O(nm) 62 Master Theorem Ernst Mayr, Harald Räcke 62

52 Example: Multiplying Two Integers A recursive approach: Suppose that integers A and B are of length n = 2 k, for some k b n 1 B B A 1 b n b n b 0 a n 1 a n B 0 A A 0 a 0 a n 2 Then it holds that A = A 1 2 n 2 + A0 and B = B 1 2 n 2 + B0 Hence, A B = A 1 B 1 2 n + (A 1 B 0 + A 0 B 1 ) 2 n 2 + A0 B 0 62 Master Theorem Ernst Mayr, Harald Räcke 63

53 Example: Multiplying Two Integers Algorithm 3 mult(a, B) 1: if A = B = 1 then 2: return a 0 b 0 3: split A into A 0 and A 1 4: split B into B 0 and B 1 5: Z 2 mult(a 1, B 1 ) 6: Z 1 mult(a 1, B 0 ) + mult(a 0, B 1 ) 7: Z 0 mult(a 0, B 0 ) 8: return Z 2 2 n + Z 1 2 n 2 + Z0 O(1) O(1) O(n) O(n) T ( n 2 ) 2T ( n 2 ) + O(n) T ( n 2 ) O(n) We get the following recurrence: ( n ) T (n) = 4T + O(n) 2 62 Master Theorem Ernst Mayr, Harald Räcke 64

54 Example: Multiplying Two Integers Master Theorem: Recurrence: T [n] = at ( n b ) + f (n) Case 1: f (n) = O(n log b a ɛ ) T (n) = Θ(n log b a ) Case 2: f (n) = Θ(n log b a log k n) T (n) = Θ(n log b a log k+1 n) Case 3: f (n) = Ω(n log b a+ɛ ) T (n) = Θ(f (n)) In our case a = 4, b = 2, and f (n) = Θ(n) Hence, we are in Case 1, since n = O(n 2 ɛ ) = O(n log b a ɛ ) We get a running time of O(n 2 ) for our algorithm Not better then the school method 62 Master Theorem Ernst Mayr, Harald Räcke 65

55 Example: Multiplying Two Integers We can use the following identity to compute Z 1 : Z 1 = A 1 B 0 + A 0 B 1 = Z 2 = Z {}}{{}} 0 { = (A 0 + A 1 ) (B 0 + B 1 ) A 1 B 1 A 0 B 0 Hence, A more precise (correct) analysis would say that computing Z 1 needs time T ( n 2 + 1) + O(n) Algorithm 4 mult(a, B) 1: if A = B = 1 then 2: return a 0 b 0 3: split A into A 0 and A 1 4: split B into B 0 and B 1 5: Z 2 mult(a 1, B 1 ) 6: Z 0 mult(a 0, B 0 ) 7: Z 1 mult(a 0 + A 1, B 0 + B 1 ) Z 2 Z 0 8: return Z 2 2 n + Z 1 2 n 2 + Z0 O(1) O(1) O(n) O(n) T ( n 2 ) T ( n 2 ) T ( n 2 ) + O(n) O(n) 62 Master Theorem Ernst Mayr, Harald Räcke 66

56 Example: Multiplying Two Integers We get the following recurrence: ( n ) T (n) = 3T + O(n) 2 Master Theorem: Recurrence: T [n] = at ( n b ) + f (n) Case 1: f (n) = O(n log b a ɛ ) T (n) = Θ(n log b a ) Case 2: f (n) = Θ(n log b a log k n) T (n) = Θ(n log b a log k+1 n) Case 3: f (n) = Ω(n log b a+ɛ ) T (n) = Θ(f (n)) Again we are in Case 1 We get a running time of Θ(n log 2 3 ) Θ(n 159 ) A huge improvement over the school method 62 Master Theorem Ernst Mayr, Harald Räcke 67

57 63 The Characteristic Polynomial Consider the recurrence relation: c 0 T (n) + c 1 T (n 1) + c 2 T (n 2) + + c k T (n k) = f (n) This is the general form of a linear recurrence relation of order k with constant coefficients (c 0, c k 0) T (n) only depends on the k preceding values This means the recurrence relation is of order k The recurrence is linear as there are no products of T [n] s If f (n) = 0 then the recurrence relation becomes a linear, homogenous recurrence relation of order k Note that we ignore boundary conditions for the moment 63 The Characteristic Polynomial Ernst Mayr, Harald Räcke 68

58 63 The Characteristic Polynomial Observations: The solution T [1], T [2], T [3], is completely determined by a set of boundary conditions that specify values for T [1],, T [k] In fact, any k consecutive values completely determine the solution k non-concecutive values might not be an appropriate set of boundary conditions (depends on the problem) Approach: First determine all solutions that satisfy recurrence relation Then pick the right one by analyzing boundary conditions First consider the homogenous case 63 The Characteristic Polynomial Ernst Mayr, Harald Räcke 69

59 The Homogenous Case The solution space { S = T = T [1], T [2], T [3], } T fulfills recurrence relation is a vector space This means that if T 1, T 2 S, then also αt 1 + βt 2 S, for arbitrary constants α, β How do we find a non-trivial solution? We guess that the solution is of the form λ n, λ 0, and see what happens In order for this guess to fulfill the recurrence we need c 0 λ n + c 1 λ n 1 + c 2 λ n c k λ n k = 0 for all n k 63 The Characteristic Polynomial Ernst Mayr, Harald Räcke 70

60 The Homogenous Case Dividing by λ n k gives that all these constraints are identical to c 0 λ k + c 1 λ k 1 + c 2 λ k c k = 0 }{{} characteristic polynomial P[λ] This means that if λ i is a root (Nullstelle) of P[λ] then T [n] = λ n i is a solution to the recurrence relation Let λ 1,, λ k be the k (complex) roots of P[λ] Then, because of the vector space property α 1 λ n 1 + α 2λ n α kλ n k is a solution for arbitrary values α i 63 The Characteristic Polynomial Ernst Mayr, Harald Räcke 71

61 The Homogenous Case Lemma 5 Assume that the characteristic polynomial has k distinct roots λ 1,, λ k Then all solutions to the recurrence relation are of the form α 1 λ n 1 + α 2λ n α kλ n k Proof There is one solution for every possible choice of boundary conditions for T [1],, T [k] We show that the above set of solutions contains one solution for every choice of boundary conditions 63 The Characteristic Polynomial Ernst Mayr, Harald Räcke 72

62 The Homogenous Case Proof (cont) Suppose I am given boundary conditions T [i] and I want to see whether I can choose the α is such that these conditions are met: α 1 λ 1 + α 2 λ α k λ k = T [1] α 1 λ α 2 λ α k λ 2 k = T [2] α 1 λ k 1 + α 2 λ k α k λ k k = T [k] 63 The Characteristic Polynomial Ernst Mayr, Harald Räcke 73

63 The Homogenous Case Proof (cont) Suppose I am given boundary conditions T [i] and I want to see whether I can choose the α is such that these conditions are met: λ 1 λ 2 λ k λ 2 1 λ 2 2 λ 2 k λ k 1 λ k 2 λ k k α 1 α 2 α k = T [1] T [2] T [k] We show that the column vectors are linearly independent Then the above equation has a solution 63 The Characteristic Polynomial Ernst Mayr, Harald Räcke 74

64 Computing the Determinant λ 1 λ 2 λ k 1 λ k λ 2 1 λ 2 2 λ 2 k 1 λ 2 k λ k 1 λ k 2 λ k k 1 λ k k = k i=1 λ i λ 1 λ 2 λ k 1 λ k λ k 1 1 λ k 1 2 λ k 1 k 1 λ k 1 k = k i=1 λ i 1 λ 1 λ k 2 1 λ k λ 2 λ k 2 2 λ k λ k λ k 2 k λ k 1 k 63 The Characteristic Polynomial Ernst Mayr, Harald Räcke 75

65 Computing the Determinant 1 λ 1 λ k 2 1 λ 2 λ k 2 1 λ k λ k 2 1 λ k λ k 1 2 k λ k 1 k = 1 λ 1 λ 1 1 λ1 k 2 λ 1 λ k 3 1 λ 2 λ 1 1 λ2 k 2 λ 1 λ k 3 1 λ k λ 1 1 λ k 2 λ 1 λ k 3 k k 1 λ k 1 2 λ k 1 1 λ 1 λ k λ 1 λ k 2 2 λ 1 λ k 2 λ k 1 k k 63 The Characteristic Polynomial Ernst Mayr, Harald Räcke 76

66 Computing the Determinant 1 λ 1 λ 1 1 λ k 2 1 λ 1 λ k 3 1 λ 2 λ 1 1 λ k 2 2 λ 1 λ k 3 1 λ k λ 1 1 λ k 2 k λ 1 λ k 3 k 1 λ k 1 2 λ k 1 1 λ 1 λ k λ 1 λ k 2 2 = λ 1 λ k 2 λ k 1 k (λ 2 λ 1 ) 1 (λ 2 λ 1 ) λ2 k 3 (λ 2 λ 1 ) λ k (λ k λ 1 ) 1 (λ k λ 1 ) λ k 3 (λ k λ 1 ) λ k 2 k k k 63 The Characteristic Polynomial Ernst Mayr, Harald Räcke 77

67 Computing the Determinant (λ 2 λ 1 ) 1 (λ 2 λ 1 ) λ k 3 2 (λ 2 λ 1 ) λ k (λ k λ 1 ) 1 (λ k λ 1 ) λ k 3 k (λ k λ 1 ) λ k 2 k = k i=2 (λ i λ 1 ) 1 λ 2 λ k 3 2 λ k λ k λ k 3 k λ k 2 k 63 The Characteristic Polynomial Ernst Mayr, Harald Räcke 78

68 Computing the Determinant Repeating the above steps gives: λ 1 λ 2 λ k 1 λ k λ 2 1 λ 2 2 λ 2 k 1 λ 2 k = λ k 1 λ k 2 λ k k 1 λ k k k λ i (λ i λ l ) Hence, if all λ i s are different, then the determinant is non-zero i=1 i>l 63 The Characteristic Polynomial Ernst Mayr, Harald Räcke 79

69 The Homogeneous Case What happens if the roots are not all distinct? Suppose we have a root λ i with multiplicity (Vielfachheit) at least 2 Then not only is λ n i a solution to the recurrence but also nλn i To see this consider the polynomial P[λ] λ n k = c 0 λ n + c 1 λ n 1 + c 2 λ n c k λ n k Since λ i is a root we can write this as Q[λ] (λ λ i ) 2 Calculating the derivative gives a polynomial that still has root λ i 63 The Characteristic Polynomial Ernst Mayr, Harald Räcke 80

70 This means c 0 nλ n 1 i + c 1 (n 1)λ n 2 i + + c k (n k)λ n k 1 i = 0 Hence, c 0 nλ n i + c 1(n 1)λ n 1 i + + c k (n k)λ n k i = 0 }{{}}{{}}{{} T [n] T [n 1] T [n k] 63 The Characteristic Polynomial Ernst Mayr, Harald Räcke 81

71 The Homogeneous Case Suppose λ i has multiplicity j We know that c 0 nλ n i + c 1(n 1)λ n 1 i + + c k (n k)λ n k i = 0 (after taking the derivative; multiplying with λ; plugging in λ i ) Doing this again gives c 0 n 2 λ n i + c 1(n 1) 2 λ n 1 i + + c k (n k) 2 λ n k i = 0 We can continue j 1 times Hence, n l λ n i is a solution for l 0,, j 1 63 The Characteristic Polynomial Ernst Mayr, Harald Räcke 82

72 The Homogeneous Case Lemma 6 Let P[λ] denote the characteristic polynomial to the recurrence c 0 T [n] + c 1 T [n 1] + + c k T [n k] = 0 Let λ i, i = 1,, m be the (complex) roots of P[λ] with multiplicities l i Then the general solution to the recurrence is given by T [n] = m l i 1 i=1 j=0 α ij (n j λ n i ) The full proof is omitted We have only shown that any choice of α ij s is a solution to the recurrence 63 The Characteristic Polynomial Ernst Mayr, Harald Räcke 83

73 Example: Fibonacci Sequence T [0] = 0 T [1] = 1 T [n] = T [n 1] + T [n 2] for n 2 The characteristic polynomial is Finding the roots, gives λ 2 λ 1 λ 1/2 = 1 2 ± = 1 ( 1 ± ) The Characteristic Polynomial Ernst Mayr, Harald Räcke 84

74 Example: Fibonacci Sequence Hence, the solution is of the form ( ) n ( β ) n α 2 2 T [0] = 0 gives α + β = 0 T [1] = 1 gives ( ) ( ) α + β 2 2 = 1 α β = The Characteristic Polynomial Ernst Mayr, Harald Räcke 85

75 Example: Fibonacci Sequence Hence, the solution is [( ) n ( ) n ] The Characteristic Polynomial Ernst Mayr, Harald Räcke 86

76 The Inhomogeneous Case Consider the recurrence relation: c 0 T (n) + c 1 T (n 1) + c 2 T (n 2) + + c k T (n k) = f (n) with f (n) 0 While we have a fairly general technique for solving homogeneous, linear recurrence relations the inhomogeneous case is different 63 The Characteristic Polynomial Ernst Mayr, Harald Räcke 87

77 The Inhomogeneous Case The general solution of the recurrence relation is T (n) = T h (n) + T p (n), where T h is any solution to the homogeneous equation, and T p is one particular solution to the inhomogeneous equation There is no general method to find a particular solution 63 The Characteristic Polynomial Ernst Mayr, Harald Räcke 88

78 The Inhomogeneous Case Example: Then, T [n] = T [n 1] + 1 T [0] = 1 T [n 1] = T [n 2] + 1 (n 2) Subtracting the first from the second equation gives, T [n] T [n 1] = T [n 1] T [n 2] (n 2) or T [n] = 2T [n 1] T [n 2] (n 2) I get a completely determined recurrence if I add T [0] = 1 and T [1] = 2 63 The Characteristic Polynomial Ernst Mayr, Harald Räcke 89

79 The Inhomogeneous Case Example: Characteristic polynomial: Then the solution is of the form λ } 2 {{ 2λ + 1 } = 0 (λ 1) 2 T [n] = α1 n + βn1 n = α + βn T [0] = 1 gives α = 1 T [1] = 2 gives 1 + β = 2 β = 1 63 The Characteristic Polynomial Ernst Mayr, Harald Räcke 90

80 The Inhomogeneous Case If f (n) is a polynomial of degree r this method can be applied r + 1 times to obtain a homogeneous equation: T [n] = T [n 1] + n 2 Shift: T [n 1] = T [n 2] + (n 1) 2 = T [n 2] + n 2 2n + 1 Difference: T [n] T [n 1] = T [n 1] T [n 2] + 2n 1 T [n] = 2T [n 1] T [n 2] + 2n 1

81 T [n] = 2T [n 1] T [n 2] + 2n 1 Shift: T [n 1] = 2T [n 2] T [n 3] + 2(n 1) 1 = 2T [n 2] T [n 3] + 2n 3 Difference: T [n] T [n 1] =2T [n 1] T [n 2] + 2n 1 2T [n 2] + T [n 3] 2n + 3 and so on T [n] = 3T [n 1] 3T [n 2] + T [n 3] + 2

82 64 Generating Functions Definition 7 (Generating Function) Let (a n ) n 0 be a sequence The corresponding generating function (Erzeugendenfunktion) is F(z) := a n z n ; n 0 exponential generating function (exponentielle Erzeugendenfunktion) is F(z) = n 0 a n n! zn 64 Generating Functions Ernst Mayr, Harald Räcke 93

83 64 Generating Functions Example 8 1 The generating function of the sequence (1, 0, 0, ) is F(z) = 1 2 The generating function of the sequence (1, 1, 1, ) is F(z) = 1 1 z 64 Generating Functions Ernst Mayr, Harald Räcke 94

84 64 Generating Functions There are two different views: A generating function is a formal power series (formale Potenzreihe) Then the generating function is an algebraic object Let f = n 0 a n z n and g = n 0 b n z n Equality: f and g are equal if a n = b n for all n Addition: f + g := n 0(a n + b n )z n Multiplication: f g := n 0 c n z n with c n = n p=0 a p b n p There are no convergence issues here 64 Generating Functions Ernst Mayr, Harald Räcke 95

85 64 Generating Functions The arithmetic view: We view a power series as a function f : C C Then, it is important to think about convergence/convergence radius etc 64 Generating Functions Ernst Mayr, Harald Räcke 96

86 64 Generating Functions What does n 0 z n = 1 1 z mean in the algebraic view? It means that the power series 1 z and the power series n 0 z n are invers, ie, ( ) ( 1 z n 0 z n) = 1 This is well-defined 64 Generating Functions Ernst Mayr, Harald Räcke 97

87 64 Generating Functions Suppose we are given the generating function n 0 We can compute the derivative: z n = 1 1 z nz n 1 = n 1 }{{} n 0(n+1)z n 1 (1 z) 2 Formally the derivative of a formal power series n 0 a n z n is defined as n 0 na n z n 1 The known rules for differentiation work for this definition In particular, eg the derivative of 1 z 1 is 1 (1 z) 2 Note that this requires a proof if we consider power series as algebraic objects However, we did not prove this in the lecture Hence, the generating function of the sequence a n = n + 1 is 1/(1 z) 2 64 Generating Functions Ernst Mayr, Harald Räcke 98

88 64 Generating Functions We can repeat this (n + 1)z n 1 = (1 z) 2 n 0 Derivative: n(n + 1)z n 1 = n 1 n 0(n+1)(n+2)z n }{{} 2 (1 z) 3 Hence, the generating function of the sequence a n = (n + 1)(n + 2) is 2 (1 z) 3 64 Generating Functions Ernst Mayr, Harald Räcke 99

89 64 Generating Functions Computing the k-th derivative of z n n(n 1) (n k + 1)z n k = n k n 0 k! = (1 z) k+1 (n + k) (n + 1)z n Hence: ( ) n + k z n 1 = k (1 z) k+1 n 0 The generating function of the sequence a n = ( ) n+k k is 1 (1 z) k+1 64 Generating Functions Ernst Mayr, Harald Räcke 100

90 64 Generating Functions nz n = + 1)z n 0 n 0(n n n 0 1 = (1 z) z z = (1 z) 2 z n The generating function of the sequence a n = n is z (1 z) 2 64 Generating Functions Ernst Mayr, Harald Räcke 101

91 64 Generating Functions We know n 0 y n = 1 1 y Hence, a n z n 1 = 1 az n 0 The generating function of the sequence f n = a n is 1 1 az 64 Generating Functions Ernst Mayr, Harald Räcke 102

92 Example: a n = a n 1 + 1, a 0 = 1 Suppose we have the recurrence a n = a n for n 1 and a 0 = 1 A(z) = a n z n n 0 = a 0 + (a n 1 + 1)z n n 1 = 1 + z a n 1 z n 1 + n 1 n 1 = z a n z n + n 0 n 0 = za(z) + n 0 = za(z) z z n z n z n 64 Generating Functions Ernst Mayr, Harald Räcke 103

93 Example: a n = a n 1 + 1, a 0 = 1 Solving for A(z) gives a n z n 1 = A(z) = (1 z) n 0 2 = (n + 1)z n n 0 Hence, a n = n Generating Functions Ernst Mayr, Harald Räcke 104

94 Some Generating Functions n-th sequence element 1 n + 1 ( n+k k n generating function 1 1 z 1 (1 z) 2 ) 1 (1 z) k+1 z (1 z) 2 a n 1 1 az n 2 z(1 + z) (1 z) 3 1 n! e z 64 Generating Functions Ernst Mayr, Harald Räcke 105

95 Some Generating Functions n-th sequence element cf n f n + g n n i=0 f ig n i generating function cf F + G F G f n k (n k); 0 otw z k F n i=0 f i nf n c n f n F(z) 1 z z df(z) dz F(cz) 64 Generating Functions Ernst Mayr, Harald Räcke 106

96 Solving Recursions with Generating Functions 1 Set A(z) = n 0 a n z n 2 Transform the right hand side so that boundary condition and recurrence relation can be plugged in 3 Do further transformations so that the infinite sums on the right hand side can be replaced by A(z) 4 Solving for A(z) gives an equation of the form A(z) = f (z), where hopefully f (z) is a simple function 5 Write f (z) as a formal power series Techniques: partial fraction decomposition (Partialbruchzerlegung) lookup in tables 6 The coefficients of the resulting power series are the a n 64 Generating Functions Ernst Mayr, Harald Räcke 107

97 Example: a n = 2a n 1, a 0 = 1 1 Set up generating function: A(z) = a n z n n 0 2 Transform right hand side so that recurrence can be plugged in: A(z) = a 0 + a n z n n 1 2 Plug in: A(z) = 1 + (2a n 1 )z n n 1 64 Generating Functions Ernst Mayr, Harald Räcke 108

98 Example: a n = 2a n 1, a 0 = 1 3 Transform right hand side so that infinite sums can be replaced by A(z) or by simple function A(z) = 1 + (2a n 1 )z n n 1 = 1 + 2z a n 1 z n 1 n 1 = 1 + 2z a n z n n 0 = 1 + 2z A(z) 4 Solve for A(z) A(z) = 1 1 2z 64 Generating Functions Ernst Mayr, Harald Räcke 109

99 Example: a n = 2a n 1, a 0 = 1 5 Rewrite f (z) as a power series: a n z n 1 = A(z) = 1 2z = 2 n z n n 0 n 0 64 Generating Functions Ernst Mayr, Harald Räcke 110

100 Example: a n = 3a n 1 + n, a 0 = 1 1 Set up generating function: A(z) = a n z n n 0 64 Generating Functions Ernst Mayr, Harald Räcke 111

101 Example: a n = 3a n 1 + n, a 0 = 1 2/3 Transform right hand side: A(z) = a n z n n 0 = a 0 + a n z n n 1 = 1 + (3a n 1 + n)z n n 1 = 1 + 3z a n 1 z n 1 + nz n n 1 n 1 = 1 + 3z a n z n + nz n n 0 n 0 z = 1 + 3zA(z) + (1 z) 2 64 Generating Functions Ernst Mayr, Harald Räcke 112

102 Example: a n = 3a n 1 + n, a 0 = 1 4 Solve for A(z): A(z) = 1 + 3zA(z) + z (1 z) 2 gives A(z) = (1 z)2 + z (1 3z)(1 z) 2 = z 2 z + 1 (1 3z)(1 z) 2 64 Generating Functions Ernst Mayr, Harald Räcke 113

103 Example: a n = 3a n 1 + n, a 0 = 1 5 Write f (z) as a formal power series: We use partial fraction decomposition: z 2 z + 1 (1 3z)(1 z) 2! = A 1 3z + B 1 z + C (1 z) 2 This gives z 2 z + 1 = A(1 z) 2 + B(1 3z)(1 z) + C(1 3z) = A(1 2z + z 2 ) + B(1 4z + 3z 2 ) + C(1 3z) = (A + 3B)z 2 + ( 2A 4B 3C)z + (A + B + C) 64 Generating Functions Ernst Mayr, Harald Räcke 114

104 Example: a n = 3a n 1 + n, a 0 = 1 5 Write f (z) as a formal power series: This leads to the following conditions: A + B + C = 1 2A + 4B + 3C = 1 A + 3B = 1 which gives A = 7 4 B = 1 4 C = Generating Functions Ernst Mayr, Harald Räcke 115

105 Example: a n = 3a n 1 + n, a 0 = 1 5 Write f (z) as a formal power series: A(z) = z z (1 z) 2 = n z n 1 4 z n 1 2 (n + 1)z n n 0 n 0 = ( 7 4 3n ) 2 (n + 1) z n n 0 = ( 7 4 3n 1 2 n 3 z 4) n n 0 n 0 6 This means a n = 7 4 3n 1 2 n Generating Functions Ernst Mayr, Harald Räcke 116

106 65 Transformation of the Recurrence Example 9 f 0 = 1 f 1 = 2 f n = f n 1 f n 2 for n 2 Define Then g n := log f n g n = g n 1 + g n 2 for n 2 g 1 = log 2 = 1(for log = log 2 ), g 0 = 0 g n = F n (n-th Fibonacci number) f n = 2 F n 65 Transformation of the Recurrence Ernst Mayr, Harald Räcke 117

107 65 Transformation of the Recurrence Example 10 f 1 = 1 f n = 3f n 2 + n; for n = 2k, k 1 ; Define g k := f 2 k Then: g 0 = 1 g k = 3g k k, k 1 65 Transformation of the Recurrence Ernst Mayr, Harald Räcke 118

108 6 Recurrences We get g k = 3 [ ] g k k [ = 3 3g k k 1] + 2 k = 3 2 [ g k 2 ] + 32 k k = 3 2 [ 3g k k 2] + 32 k k = 3 3 g k k k k = 2 k k ( 3 i 2) i=0 = 2 k ( 3 2 )k+1 1 1/2 = 3 k+1 2 k+1 65 Transformation of the Recurrence Ernst Mayr, Harald Räcke 119

109 6 Recurrences Let n = 2 k : g k = 3 k+1 2 k+1, hence f n = 3 3 k 2 2 k = 3(2 log 3 ) k 2 2 k = 3(2 k ) log k = 3n log 3 2n 65 Transformation of the Recurrence Ernst Mayr, Harald Räcke 120

110 6 Recurrences Bibliography [MS08] [CLRS90] [Liu85] Kurt Mehlhorn, Peter Sanders: Algorithms and Data Structures The Basic Toolbox, Springer, 2008 Thomas H Cormen, Charles E Leiserson, Ron L Rivest, Clifford Stein: Introduction to algorithms (3rd ed), MIT Press and McGraw-Hill, 2009 Chung Laung Liu: Elements of Discrete Mathematics McGraw-Hill, 1985 The Karatsuba method can be found in [MS08] Chapter 1 Chapter 43 of [CLRS90] covers the Substitution method which roughly corresponds to Guessing+induction Chapters 44, 45, 46 of this book cover the master theorem Methods using the characteristic polynomial and generating functions can be found in [Liu85] Chapter Transformation of the Recurrence Ernst Mayr, Harald Räcke 121