WS 3/4 Efficient Algorithms and Data Structures Harald Räcke Part I Organizational Matters Fakultät für Informatik TU München http://www4.in.tum.de/lehre/3ws/ea/ Winter Term 3/4 Ernst Mayr, Harald Räcke Ernst Mayr, Harald Räcke Modul: IN3 Part I Organizational Matters Name: Efficient Algorithms and Data Structures Effiziente Algorithmen und Datenstrukturen ECTS: 8 Credit points Lectures: 4 SWS Mon :5 :45 (Room Interim) Fri :5 :45 (Room Interim) Webpage: http://www4.in.tum.de/lehre/3ws/ea/ Required knowledge: IN, IN3 Introduction to Informatics / Einführung in die Informatik / IN7 Fundamentals of Algorithms and Data Structures Grundlagen: Algorithmen und Datenstrukturen (GAD) IN Basic Theoretic Informatics Einführung in die Theoretische Informatik (THEO) IN5 Discrete Structures Diskrete Strukturen (DS) IN8 Discrete Probability Theory Diskrete Wahrscheinlichkeitstheorie (DWT) Ernst Mayr, Harald Räcke 4
The Lecturer Tutorials Harald Räcke Email: raecke@in.tum.de Room: 3.9.44 Office hours: (per appointment) Tutors: Chintan Shah chintan.shah@tum.de Room: 3.9.59 Office hours: Wed :3 :3 Richard Stotz richardstotz@gmail.com Room: Office hours: Ernst Mayr, Harald Räcke 5 Ernst Mayr, Harald Räcke 6 Tutorials Assignment sheets Monday 6-8 (MI.8.38) Chintan Tuesday 4-6 (MI.8.38) Richard Thursday - (MI.8.38) Richard Friday -4 (MI.3.9A) Chintan In order to pass the module you need to pass a 3 hour exam. Ernst Mayr, Harald Räcke 7 Ernst Mayr, Harald Räcke 8
Assessment Assignment Sheets: An assignment sheet is usually made available on Monday on the module webpage. Solutions have to be handed in in the following week before the lecture on Monday. You can hand in your solutions by putting them in the right folder in front of room 3.9.5. Solutions have to be given in English. Solutions will be discussed in the tutorial of the week when the sheet has been handed in, i.e, sheet may not be corrected by this time. You can submit solutions in groups of up to 3 people. Assessment Assignment can be used to improve you grade If you obtain 5% of the points on the first half and 5% on the second half of assignments your grade will improve according to the following function ( )) f (x) = ( round round(3x) 3 < x < 4 x otw. It will improve by.3 or.4, respectively. Examples: 3.3 3...7 3.7 3.3.. > 4. no improvement Ernst Mayr, Harald Räcke 9 Ernst Mayr, Harald Räcke Contents Literatur Foundations Machine models Efficiency measures Asymptotic notation Recursion Higher Data Structures Search trees Hashing Priority queues Union/Find data structures Cuts/Flows Matchings Alfred V. Aho, John E. Hopcroft, Jeffrey D. Ullman: The design and analysis of computer algorithms, Addison-Wesley Publishing Company: Reading (MA), 974 Thomas H. Cormen, Charles E. Leiserson, Ron L. Rivest, Clifford Stein: Introduction to algorithms, McGraw-Hill, 99 Michael T. Goodrich, Roberto Tamassia: Algorithm design: Foundations, analysis, and internet examples, John Wiley & Sons, Contents Ernst Mayr, Harald Räcke Literatur Ernst Mayr, Harald Räcke
Literatur Volker Heun: Grundlegende Algorithmen: Einführung in den Entwurf und die Analyse effizienter Algorithmen,. Auflage, Vieweg, 3 Jon Kleinberg, Eva Tardos: Algorithm Design, Addison-Wesley, 5 Donald E. Knuth: The art of computer programming. Vol. : Fundamental Algorithms, 3. Auflage, Addison-Wesley Publishing Company: Reading (MA), 997 Literatur Donald E. Knuth: The art of computer programming. Vol. 3: Sorting and Searching, 3. Auflage, Addison-Wesley Publishing Company: Reading (MA), 997 Christos H. Papadimitriou, Kenneth Steiglitz: Combinatorial Optimization: Algorithms and Complexity, Prentice Hall, 98 Uwe Schöning: Algorithmik, Spektrum Akademischer Verlag, Steven S. Skiena: The Algorithm Design Manual, Springer, 998 Literatur Ernst Mayr, Harald Räcke 3 Literatur Ernst Mayr, Harald Räcke 4 Vocabularies Part II Foundations a b a times b a multiplied by b a into b a b a b a divided by b a by b a over b (a: numerator (Zähler), b: denominator (Nenner)) a raised to the b-th power a to the b-th a raised to the power of b a to the power of b a raised to b a to the b a raised by the exponent of b Ernst Mayr, Harald Räcke 5 Ernst Mayr, Harald Räcke 6
Vocabularies 3 Goals n! n factorial ( n k) n choose k x i x subscript i x sub i x i log b a log to the base b of a log a to the base b Gain knowledge about efficient algorithms for important problems, i.e., learn how to solve certain types of problems efficiently. Learn how to analyze and judge the efficiency of algorithms. Learn how to design efficient algorithms. f : X Y, x x f is a function that maps from domain (Definitionsbereich) X to codomain (Zielmenge) Y. The set {y Y x X : f (x) = y} is the image or the range of the function (Bildbereich/Wertebereich). Ernst Mayr, Harald Räcke 7 3 Goals Ernst Mayr, Harald Räcke 8 4 Modelling Issues 4 Modelling Issues How do you measure? What do you measure? Memory requirement Running time Number of comparisons Number of multiplications Number of hard-disc accesses Program size Power consumption... Implementing and testing on representative inputs How do you choose your inputs? May be very time-consuming. Very reliable results if done correctly. Results only hold for a specific machine and for a specific set of inputs. Theoretical analysis in a specific model of computation. Gives asymptotic bounds like this algorithm always runs in time O(n ). Typically focuses on the worst case. Can give lower bounds like any comparison-based sorting algorithm needs at least Ω(n log n) comparisons in the worst case. 4 Modelling Issues Ernst Mayr, Harald Räcke 9 4 Modelling Issues Ernst Mayr, Harald Räcke
4 Modelling Issues Model of Computation Input length The theoretical bounds are usually given by a function f : N N that maps the input length to the running time (or storage space, comparisons, multiplications, program size etc.). The input length may e.g. be the size of the input (number of bits) the number of arguments Example Suppose n numbers from the interval {,..., N} have to be sorted. In this case we usually say that the input length is n instead of e.g. n log N, which would be the number of bits required to encode the input. How to measure performance. Calculate running time and storage space etc. on a simplified, idealized model of computation, e.g. Random Access Machine (RAM), Turing Machine (TM),.... Calculate number of certain basic operations: comparisons, multiplications, harddisc accesses,... Version. is often easier, but focusing on one type of operation makes it more difficult to obtain meaningful results. 4 Modelling Issues Ernst Mayr, Harald Räcke 4 Modelling Issues Ernst Mayr, Harald Räcke Turing Machine Very simple model of computation. Only the current memory location can be altered. Very good model for discussing computabiliy, or polynomial vs. exponential time. Some simple problems like recognizing whether input is of the form xx, where x is a string, have quadratic lower bound. Not a good model for developing efficient algorithms....... Random Access Machine (RAM) Input tape and output tape (sequences of zeros and ones; unbounded length). Memory unit: infinite but countable number of registers R[], R[], R[],.... Registers hold integers. Indirect addressing. input tape...... control unit memory R[] R[] R[] R[3] R[4] control unit state state holds program and can act as constant size memory Note that in the picture on the right the tapes are one-directional, and that a READ- or WRITE-operation always advances its tape....... output tape R[5]. 4 Modelling Issues Ernst Mayr, Harald Räcke 3 4 Modelling Issues Ernst Mayr, Harald Räcke 4
Random Access Machine (RAM) Operations input operations (input tape R[i]) READ i output operations (R[i] output tape) WRITE i register-register transfers R[j] := R[i] R[j] := 4 indirect addressing R[j] := R[R[i]] loads the content of the R[i]-th register into the j-th register R[R[i]] := R[j] loads the content of the j-th into the R[i]-th register Random Access Machine (RAM) Operations branching (including loops) based on comparisons jump x jumps to position x in the program; sets instruction counter to x; reads the next operation to perform from register R[x] jumpz x R[i] jump to x if R[i] = if not the instruction counter is increased by ; jumpi i jump to R[i] (indirect jump); arithmetic instructions: +,,, / R[i] := R[j] + R[k]; R[i] := -R[k]; The jump-directives are very close to the jump-instructions contained in the assembler language of real machines. 4 Modelling Issues Ernst Mayr, Harald Räcke 5 4 Modelling Issues Ernst Mayr, Harald Räcke 6 Model of Computation uniform cost model Every operation takes time. logarithmic cost model The cost depends on the content of memory cells: The time for a step is equal to the largest operand involved; The storage space of a register is equal to the length (in bits) of the largest value ever stored in it. Bounded word RAM model: cost is uniform but the largest value stored in a register may not exceed w, where usually w = log n. The latter model is quite realistic as the word-size of a standard computer that handles a problem of size n must be at least log n as otherwise the computer could either not store the problem instance or not address all its memory. 4 Modelling Issues Example Algorithm RepeatedSquaring(n) : r ; : for i = n do 3: r r 4: return r running time: uniform model: n steps logarithmic model: + + 4 + + n = n+ = Θ( n ) space requirement: uniform model: O() logarithmic model: O( n ) 4 Modelling Issues Ernst Mayr, Harald Räcke 7 4 Modelling Issues Ernst Mayr, Harald Räcke 8
There are different types of complexity bounds: best-case complexity: C bc (n) := min{c(x) x = n} Usually easy to analyze, but not very meaningful. worst-case complexity: C wc (n) := max{c(x) x = n} Usually moderately easy to analyze; sometimes too pessimistic. average case complexity: C avg (n) := I n x =n more general: probability measure µ C avg (n) := C(x) x I n µ(x) C(x) cost of instance C(x) x x input length of instance x I n set of instances of length n There are different types of complexity bounds: amortized complexity: The average cost of data structure operations over a worst case sequence of operations. randomized complexity: The algorithm may use random bits. Expected running time (over all possible choices of random bits) for a fixed input x. Then take the worst-case over all x with x = n. 4 Modelling Issues Ernst Mayr, Harald Räcke 9 4 Modelling Issues Ernst Mayr, Harald Räcke 9 4 Modelling Issues Bibliography [MS8] [CLRS9] Kurt Mehlhorn, Peter Sanders: Algorithms and Data Structures The Basic Toolbox, Springer, 8 Thomas H. Cormen, Charles E. Leiserson, Ron L. Rivest, Clifford Stein: Introduction to algorithms (3rd ed.), McGraw-Hill, 9 Chapter. and. of [MS8] and Chapter of [CLRS9] are relevant for this section. 5 Asymptotic Notation We are usually not interested in exact running times, but only in an asymptotic classification of the running time, that ignores constant factors and constant additive offsets. We are usually interested in the running times for large values of n. Then constant additive terms do not play an important role. An exact analysis (e.g. exactly counting the number of operations in a RAM) may be hard, but wouldn t lead to more precise results as the computational model is already quite a distance from reality. A linear speed-up (i.e., by a constant factor) is always possible by e.g. implementing the algorithm on a faster machine. Running time should be expressed by simple functions. 4 Modelling Issues Ernst Mayr, Harald Räcke 3 5 Asymptotic Notation Ernst Mayr, Harald Räcke 3
Asymptotic Notation Formal Definition Let f denote functions from N to R +. O(f ) = {g c > n N n n : [g(n) c f (n)]} (set of functions that asymptotically grow not faster than f ) Ω(f ) = {g c > n N n n : [g(n) c f (n)]} (set of functions that asymptotically grow not slower than f ) Θ(f ) = Ω(f ) O(f ) (functions that asymptotically have the same growth as f ) o(f ) = {g c > n N n n : [g(n) c f (n)]} (set of functions that asymptotically grow slower than f ) ω(f ) = {g c > n N n n : [g(n) c f (n)]} (set of functions that asymptotically grow faster than f ) Asymptotic Notation There is an equivalent definition using limes notation (assuming that the respective limes exists). f and g are functions from N to R +. g O(f ): g Ω(f ): g Θ(f ): g o(f ): g ω(f ): lim n g(n) f (n) < < lim n g(n) f (n) < lim n g(n) f (n) < g(n) lim n f (n) = g(n) lim n f (n) = Note that for the version of the Landau notation defined here, we assume that f and g are positive functions. There also exist versions for arbitrary functions, and for the case that the limes is not infinity. 5 Asymptotic Notation Ernst Mayr, Harald Räcke 3 5 Asymptotic Notation Ernst Mayr, Harald Räcke 3 Asymptotic Notation Abuse of notation. People write f = O(g), when they mean f O(g). This is not an equality (how could a function be equal to a set of functions).. People write f (n) = O(g(n)), when they mean f O(g), with f : N R +, n f (n), and g : N R +, n g(n). 3. People write e.g. h(n) = f (n) + o(g(n)) when they mean that there exists a function z : N R +, n z(n), z o(g) such that h(n) = f (n) + z(n). Asymptotic Notation Abuse of notation 4. People write O(f (n)) = O(g(n)), when they mean O(f (n)) O(g(n)). Again this is not an equality.. In this context f (n) does not mean the function f evaluated at n, but instead it is a shorthand for the function itself (leaving out domain and codomain and only giving the rule of correspondence of the function). 3. This is particularly useful if you do not want to ignore constant factors. For example the median of n elements can be determined using 3 n+o(n) comparisons. 5 Asymptotic Notation Ernst Mayr, Harald Räcke 33 5 Asymptotic Notation Ernst Mayr, Harald Räcke 33
Asymptotic Notation in Equations Asymptotic Notation in Equations How do we interpret an expression like: n + 3n + = n + Θ(n) Here, Θ(n) stands for an anonymous function in the set Θ(n) that makes the expression true. Note that Θ(n) is on the right hand side, otw. this interpretation is wrong. How do we interpret an expression like: n + O(n) = Θ(n ) Regardless of how we choose the anonymous function f (n) O(n) there is an anonymous function g(n) Θ(n ) that makes the expression true. 5 Asymptotic Notation Ernst Mayr, Harald Räcke 34 5 Asymptotic Notation Ernst Mayr, Harald Räcke 35 Asymptotic Notation in Equations How do we interpret an expression like: n Θ(i) = Θ(n ) i= The Θ(i)-symbol on the left represents one anonymous function f : N R +, and then i f (i) is computed. Asymptotic Notation in Equations We can view an expression containing asymptotic notation as generating a set: n O(n) + O(log n) Careful! It is understood that every occurence of an O-symbol (or Θ, Ω, o, ω) on the left represents one anonymous function. Hence, the left side is not equal to Θ() + Θ() + + Θ(n ) + Θ(n) Θ()+Θ()+ +Θ(n )+Θ(n) does not really have a reasonable interpretation. represents { f : N R + f (n) = n g(n) + h(n) } with g(n) O(n) and h(n) O(log n) Recall that according to the previous slide e.g. the expressions n i= O(i) and n/ i= O(i) + n i=n/+ O(i) generate different sets. 5 Asymptotic Notation Ernst Mayr, Harald Räcke 36 5 Asymptotic Notation Ernst Mayr, Harald Räcke 37
Asymptotic Notation in Equations Asymptotic Notation Then an asymptotic equation can be interpreted as containement btw. two sets: n O(n) + O(log n) = Θ(n ) represents n O(n) + O(log n) Θ(n ) Lemma 3 Let f, g be functions with the property n > n n : f (n) > (the same for g). Then c f (n) Θ(f (n)) for any constant c O(f (n)) + O(g(n)) = O(f (n) + g(n)) O(f (n)) O(g(n)) = O(f (n) g(n)) O(f (n)) + O(g(n)) = O(max{f (n), g(n)}) The expressions also hold for Ω. Note that this means that f (n) + g(n) Θ(max{f (n), g(n)}). Note that the equation does not hold. 5 Asymptotic Notation Ernst Mayr, Harald Räcke 38 5 Asymptotic Notation Ernst Mayr, Harald Räcke 39 Asymptotic Notation Asymptotic Notation Comments Do not use asymptotic notation within induction proofs. For any constants a, b we have log a n = Θ(log b n). Therefore, we will usually ignore the base of a logarithm within asymptotic notation. In general log n = log n, i.e., we use as the default base for the logarithm. In general asymptotic classification of running times is a good measure for comparing algorithms: If the running time analysis is tight and actually occurs in practise (i.e., the asymptotic bound is not a purely theoretical worst-case bound), then the algorithm that has better asymptotic running time will always outperform a weaker algorithm for large enough values of n. However, suppose that I have two algorithms: Algorithm A. Running time f (n) = log n = O(log n). Algorithm B. Running time g(n) = log n. Clearly f = o(g). However, as long as log n Algorithm B will be more efficient. 5 Asymptotic Notation Ernst Mayr, Harald Räcke 4 5 Asymptotic Notation Ernst Mayr, Harald Räcke 4
5 Asymptotic Notation 6 Recurrences Bibliography [MS8] [CLRS9] Kurt Mehlhorn, Peter Sanders: Algorithms and Data Structures The Basic Toolbox, Springer, 8 Thomas H. Cormen, Charles E. Leiserson, Ron L. Rivest, Clifford Stein: Introduction to algorithms (3rd ed.), McGraw-Hill, 9 Mainly Chapter 3 of [CLRS9]. [MS8] covers this topic in chapter. but not very detailed. Algorithm mergesort(list L) : n size(l) : if n return L 3: L L[ n ] 4: L L[ n + n] 5: mergesort(l ) 6: mergesort(l ) 7: L merge(l, L ) 8: return L This algorithm requires ( n ) ( n ) ( n ) T (n) = T + T + O(n) T + O(n) comparisons when n > and comparisons when n. 5 Asymptotic Notation Ernst Mayr, Harald Räcke 4 6 Recurrences Ernst Mayr, Harald Räcke 4 Recurrences Methods for Solving Recurrences How do we bring the expression for the number of comparisons ( running time) into a closed form? For this we need to solve the recurrence.. Guessing+Induction Guess the right solution and prove that it is correct via induction. It needs experience to make the right guess.. Master Theorem For a lot of recurrences that appear in the analysis of algorithms this theorem can be used to obtain tight asymptotic bounds. It does not provide exact solutions. 3. Characteristic Polynomial Linear homogenous recurrences can be solved via this method. 6 Recurrences Ernst Mayr, Harald Räcke 43 6 Recurrences Ernst Mayr, Harald Räcke 44
Methods for Solving Recurrences 6. Guessing+Induction 4. Generating Functions A more general technique that allows to solve certain types of linear inhomogenous relations and also sometimes non-linear recurrence relations. 5. Transformation of the Recurrence Sometimes one can transform the given recurrence relations so that it e.g. becomes linear and can therefore be solved with one of the other techniques. First we need to get rid of the O-notation in our recurrence: { ( T n ) + cn n T (n) otherwise Assume that instead we had { ( T n ) + cn n T (n) otherwise One way of solving such a recurrence is to guess a solution, and check that it is correct by plugging it in. 6 Recurrences Ernst Mayr, Harald Räcke 45 6. Guessing+Induction Ernst Mayr, Harald Räcke 46 6. Guessing+Induction Suppose we guess T (n) dn log n for a constant d. Then if we choose d c. ( n ) T (n) T + cn ( d n log n ) + cn = dn(log n ) + cn = dn log n + (c d)n dn log n Formally one would make an induction proof, where the above is the induction step. The base case is usually trivial. 6. Guessing+Induction Guess: T (n) dn log n. Proof. (by induction) { ( n ) T + cn n 6 T (n) b otw. base case ( n < 6): true if we choose d b. induction step... n n: Suppose statem. is true for n {,..., n }, and n 6. We prove it for n: ( n ) T (n) T + cn ( d n log n ) + cn = dn(log n ) + cn = dn log n + (c d)n dn log n Note that this proves the statement for n N, as the statement is wrong for n =. The base case is usually omitted, as it is the same for different recurrences. 6. Guessing+Induction Ernst Mayr, Harald Räcke 47 Hence, statement is true if we choose d c.
6. Guessing+Induction Why did we change the recurrence by getting rid of the ceiling? If we do not do this we instead consider the following recurrence: { T ( n ) + cn n 6 T (n) b otherwise 6. Guessing+Induction We also make a guess of T (n) dn log n and get n n + n + 9 6 n log 9 6 n = log n + (log 9 4) ( n ) T (n) T ( n d + cn log n ) + cn ( d(n/ + ) log(n/ + ) ) + cn ( 9 ) dn log 6 n + d log n + cn = dn log n + (log 9 4)dn + d log n + cn Note that we can do this as for constant-sized inputs the running time is always some constant (b in the above case). log n n 4 dn log n + (log 9 3.5)dn + cn dn log n.33dn + cn dn log n for a suitable choice of d. 6. Guessing+Induction Ernst Mayr, Harald Räcke 49 6. Guessing+Induction Ernst Mayr, Harald Räcke 5 6. Master Theorem Note that the cases do not cover all possibilities. 6. Master Theorem Lemma 4 Let a, b and ɛ > denote constants. Consider the recurrence ( n ) T (n) = at + f (n). b Case. If f (n) = O(n log b(a) ɛ ) then T (n) = Θ(n log b a ). Case. If f (n) = Θ(n log b(a) log k n) then T (n) = Θ(n log b a log k+ n), k. We prove the Master Theorem for the case that n is of the form b l, and we assume that the non-recursive case occurs for problem size and incurs cost. Case 3. If f (n) = Ω(n logb(a)+ɛ ) and for sufficiently large n af ( n b ) cf (n) for some constant c < then T (n) = Θ(f (n)). 6. Master Theorem Ernst Mayr, Harald Räcke 5 6. Master Theorem Ernst Mayr, Harald Räcke 5
The Recursion Tree 6. Master Theorem The running time of a recursive algorithm can be visualized by a recursion tree: n b a nx n b a a n b a f (n) af ( n b ) This gives log b n ( ) T (n) = n log b n a + a i f b i i=. n b n b n b n b n b a a a a a a a a a n b n b n b n b a f ( n b ) a log b n = n log b a 6. Master Theorem Ernst Mayr, Harald Räcke 53 6. Master Theorem Ernst Mayr, Harald Räcke 54 Case. Now suppose that f (n) cn log b a ɛ. T (n) n log b a = b i(log b a ɛ) = b ɛi (b log b a ) i = b ɛi a i Hence, k i= qi = qk+ q ( ) c T (n) b ɛ + log b n i= ( ) n a i f b i log b n ( ) n logb c a i a ɛ b i i= = cn log b a ɛ log b n i= ( b ɛ ) i = cn log b a ɛ (b ɛ log b n )/(b ɛ ) = cn log b a ɛ (n ɛ )/(b ɛ ) c = b ɛ nlog b a (n ɛ )/(n ɛ ) n log b(a) T (n) = O(n log b a ). Case. Now suppose that f (n) cn log b a. Hence, T (n) n log b a = log b n i= ( ) n a i f b i log b n ( ) n logb c a i a b i i= log b n = cn log b a i= = cn log b a log b n T (n) = O(n log b a log b n) T (n) = O(n log b a log n). 6. Master Theorem Ernst Mayr, Harald Räcke 55 6. Master Theorem Ernst Mayr, Harald Räcke 56
Case. Now suppose that f (n) cn log b a. Hence, T (n) n log b a = log b n i= ( ) n a i f b i log b n ( ) n logb c a i a b i i= log b n = cn log b a i= = cn log b a log b n T (n) = Ω(n log b a log b n) T (n) = Ω(n log b a log n). Case. Now suppose that f (n) cn log b a (log b (n)) k. T (n) n log b a = n = b l l = log b n log b n i= ( ) n a i f b i log b n ( ) n logb c a i a b i i= = cn log b a l i= ( ( b l )) k log b l = cn log b a (l i) k = cn log b a i= l i= i k c k nlog b a l k+ b i k lk+ ( ( )) n k log b b i T (n) = O(n log b a log k+ n). 6. Master Theorem Ernst Mayr, Harald Räcke 57 6. Master Theorem Ernst Mayr, Harald Räcke 58 Case 3. Now suppose that f (n) dn log b a+ɛ, and that for sufficiently large n: af (n/b) cf (n), for c <. From this we get a i f (n/b i ) c i f (n), where we assume that n/b i n is still sufficiently large. Hence, T (n) n log b a = q < : n i= qi = qn+ q q log b n i= log b n i= ( ) n a i f b i c i f (n) + O(n log b a ) c f (n) + O(nlog b a ) T (n) O(f (n)) T (n) = Θ(f (n)). Example: Multiplying Two Integers Suppose we want to multiply two n-bit Integers, but our registers can only perform operations on integers of constant size. For this we first need to be able to add two integers A and B: A B This gives that two n-bit integers can be added in time O(n). Where did we use f (n) Ω(n log b a+ɛ )? 6. Master Theorem Ernst Mayr, Harald Räcke 59 6. Master Theorem Ernst Mayr, Harald Räcke 6
Example: Multiplying Two Integers Suppose that we want to multiply an n-bit integer A and an m-bit integer B (m n). This is also nown as the school method for multiplying integers. Note that the intermediate numbers that are generated can have at most m + n n bits. Example: Multiplying Two Integers A recursive approach: Suppose that integers A and B are of length n = k, for some k. b n. B.. b n. B. b. n... b.... A. a. n... B A A a Then it holds that a n a n A = A n + A and B = B n + B Time requirement: Computing intermediate results: O(nm). Adding m numbers of length n: O((m + n)m) = O(nm). Hence, A B = A B n + (A B + A B ) n + A B 6. Master Theorem Ernst Mayr, Harald Räcke 6 6. Master Theorem Ernst Mayr, Harald Räcke 6 Example: Multiplying Two Integers Example: Multiplying Two Integers Algorithm 3 mult(a, B) : if A = B = then : return a b 3: split A into A and A 4: split B into B and B 5: Z mult(a, B ) 6: Z mult(a, B ) + mult(a, B ) 7: Z mult(a, B ) 8: return Z n + Z n + Z We get the following recurrence: ( n ) T (n) = 4T + O(n). O() O() O(n) O(n) T ( n ) T ( n ) + O(n) T ( n ) O(n) Master Theorem: Recurrence: T [n] = at ( n b ) + f (n). Case : f (n) = O(n log b a ɛ ) T (n) = Θ(n log b a ) Case : f (n) = Θ(n log b a log k n) T (n) = Θ(n log b a log k+ n) Case 3: f (n) = Ω(n log b a+ɛ ) T (n) = Θ(f (n)) In our case a = 4, b =, and f (n) = Θ(n). Hence, we are in Case, since n = O(n ɛ ) = O(n log b a ɛ ). We get a running time of O(n ) for our algorithm. Not better then the school method. 6. Master Theorem Ernst Mayr, Harald Räcke 63 6. Master Theorem Ernst Mayr, Harald Räcke 64
Example: Multiplying Two Integers We can use the following identity to compute Z : Hence, A more precise (correct) analysis would say that computing Z needs time T ( n + ) + O(n). Z = A B + A B = Z = Z {}}{{}} { = (A + A ) (B + B ) A B A B Algorithm 4 mult(a, B) : if A = B = then : return a b 3: split A into A and A 4: split B into B and B 5: Z mult(a, B ) 6: Z mult(a, B ) 7: Z mult(a + A, B + B ) Z Z 8: return Z n + Z n + Z O() O() O(n) O(n) T ( n ) T ( n ) T ( n ) + O(n) O(n) Example: Multiplying Two Integers We get the following recurrence: ( n ) T (n) = 3T + O(n). Master Theorem: Recurrence: T [n] = at ( n b ) + f (n). Case : f (n) = O(n log b a ɛ ) T (n) = Θ(n log b a ) Case : f (n) = Θ(n log b a log k n) T (n) = Θ(n log b a log k+ n) Case 3: f (n) = Ω(n log b a+ɛ ) T (n) = Θ(f (n)) Again we are in Case. We get a running time of Θ(n log 3 ) Θ(n.59 ). A huge improvement over the school method. 6. Master Theorem Ernst Mayr, Harald Räcke 65 6. Master Theorem Ernst Mayr, Harald Räcke 66 6.3 The Characteristic Polynomial Consider the recurrence relation: c T (n) + c T (n ) + c T (n ) + + c k T (n k) = f (n) This is the general form of a linear recurrence relation of order k with constant coefficients (c, c k ). T (n) only depends on the k preceding values. This means the recurrence relation is of order k. The recurrence is linear as there are no products of T [n] s. If f (n) = then the recurrence relation becomes a linear, homogenous recurrence relation of order k. Note that we ignore boundary conditions for the moment. 6.3 The Characteristic Polynomial Observations: The solution T [], T [], T [3],... is completely determined by a set of boundary conditions that specify values for T [],..., T [k]. In fact, any k consecutive values completely determine the solution. k non-concecutive values might not be an appropriate set of boundary conditions (depends on the problem). Approach: First determine all solutions that satisfy recurrence relation. Then pick the right one by analyzing boundary conditions. First consider the homogenous case. 6.3 The Characteristic Polynomial Ernst Mayr, Harald Räcke 67 6.3 The Characteristic Polynomial Ernst Mayr, Harald Räcke 68
The Homogenous Case The solution space { S = T = T [], T [], T [3],... } T fulfills recurrence relation is a vector space. This means that if T, T S, then also αt + βt S, for arbitrary constants α, β. How do we find a non-trivial solution? We guess that the solution is of the form λ n, λ, and see what happens. In order for this guess to fulfill the recurrence we need for all n k. c λ n + c λ n + c λ n + + c k λ n k = The Homogenous Case Dividing by λ n k gives that all these constraints are identical to c λ k + c λ k + c λ k + + c k = }{{} characteristic polynomial P[λ] This means that if λ i is a root (Nullstelle) of P[λ] then T [n] = λ n i is a solution to the recurrence relation. Let λ,..., λ k be the k (complex) roots of P[λ]. Then, because of the vector space property α λ n + α λ n + + α kλ n k is a solution for arbitrary values α i. 6.3 The Characteristic Polynomial Ernst Mayr, Harald Räcke 69 6.3 The Characteristic Polynomial Ernst Mayr, Harald Räcke 7 The Homogenous Case The Homogenous Case Lemma 5 Assume that the characteristic polynomial has k distinct roots λ,..., λ k. Then all solutions to the recurrence relation are of the form α λ n + α λ n + + α kλ n k. Proof. There is one solution for every possible choice of boundary conditions for T [],..., T [k]. Proof (cont.). Suppose I am given boundary conditions T [i] and I want to see whether I can choose the α is such that these conditions are met: α λ + α λ + + α k λ k = T [] α λ + α λ + + α k λ k = T []. α λ k + α λ k + + α k λ k k = T [k] We show that the above set of solutions contains one solution for every choice of boundary conditions. 6.3 The Characteristic Polynomial Ernst Mayr, Harald Räcke 7 6.3 The Characteristic Polynomial Ernst Mayr, Harald Räcke 7
The Homogenous Case Proof (cont.). Suppose I am given boundary conditions T [i] and I want to see whether I can choose the α is such that these conditions are met: λ λ λ k λ λ λ k. λ k λ k λ k k α α. α k = T [] T []. T [k] We show that the column vectors are linearly independent. Then the above equation has a solution. λ λ λ k λ k λ λ λ k λ k k.. = λ.. i i= λ k λ k λ k k λ k k = λ λ λ k λ k.... λ k λ k λ k λ λ k k λ λ k λ i... λ k λ k i= k λ k λ k k. λ k k λ k k 6.3 The Characteristic Polynomial Ernst Mayr, Harald Räcke 73 6.3 The Characteristic Polynomial Ernst Mayr, Harald Räcke 74 Computing the Determinant Computing the Determinant λ λ k λ λ k... λ k λ k λ k λ k k. λ k k = λ λ λ k λ λ k 3 λ λ λ k λ λ k 3... λ k λ λ k λ λ k 3 k k λ k λ k λ λ k λ λ k. λ λ k λ k k k λ λ λ k λ λ k 3 λ λ λ k λ λ k 3... λ k λ λ k k λ λ k 3 k λ k λ k λ λ k λ λ k =. λ λ k λ k k (λ λ ) (λ λ ) λ k 3 (λ λ ) λ k.... (λ k λ ) (λ k λ ) λ k 3 (λ k λ ) λ k k k k 6.3 The Characteristic Polynomial Ernst Mayr, Harald Räcke 75 6.3 The Characteristic Polynomial Ernst Mayr, Harald Räcke 76
Computing the Determinant Computing the Determinant (λ λ ) (λ λ ) λ k 3 (λ λ ) λ k.. =.. (λ k λ ) (λ k λ ) λ k 3 (λ k λ ) λ k λ k λ k 3 (λ i λ )... λ k λ k 3 i= k λ k k. λ k k k Repeating the above steps gives: λ λ λ k λ k λ λ λ k λ k.. =.. λ k λ k λ k k λ k k k λ i (λ i λ l ) Hence, if all λ i s are different, then the determinant is non-zero. i= i>l 6.3 The Characteristic Polynomial Ernst Mayr, Harald Räcke 77 6.3 The Characteristic Polynomial Ernst Mayr, Harald Räcke 78 The Homogeneous Case What happens if the roots are not all distinct? Suppose we have a root λ i with multiplicity (Vielfachheit) at least. Then not only is λ n i a solution to the recurrence but also nλn i. To see this consider the polynomial P[λ] λ n k = c λ n + c λ n + c λ n + + c k λ n k Since λ i is a root we can write this as Q[λ] (λ λ i ). Calculating the derivative gives a polynomial that still has root λ i. This means Hence, c nλ n i + c (n )λ n i + + c k (n k)λ n k i = c nλ n i + c (n )λ n i + + c k (n k)λ n k i = }{{}}{{}}{{} T [n] T [n ] T [n k] 6.3 The Characteristic Polynomial Ernst Mayr, Harald Räcke 79 6.3 The Characteristic Polynomial Ernst Mayr, Harald Räcke 8
The Homogeneous Case The Homogeneous Case Suppose λ i has multiplicity j. We know that c nλ n i + c (n )λ n i + + c k (n k)λ n k i = (after taking the derivative; multiplying with λ; plugging in λ i ) Doing this again gives c n λ n i + c (n ) λ n i We can continue j times. + + c k (n k) λ n k i = Lemma 6 Let P[λ] denote the characteristic polynomial to the recurrence c T [n] + c T [n ] + + c k T [n k] = Let λ i, i =,..., m be the (complex) roots of P[λ] with multiplicities l i. Then the general solution to the recurrence is given by T [n] = m l i i= j= α ij (n j λ n i ). Hence, n l λ n i is a solution for l,..., j. The full proof is omitted. We have only shown that any choice of α ij s is a solution to the recurrence. 6.3 The Characteristic Polynomial Ernst Mayr, Harald Räcke 8 6.3 The Characteristic Polynomial Ernst Mayr, Harald Räcke 8 Example: Fibonacci Sequence Example: Fibonacci Sequence T [] = T [] = T [n] = T [n ] + T [n ] for n Hence, the solution is of the form ( ) n ( + 5 5 α + β ) n The characteristic polynomial is Finding the roots, gives λ λ λ / = ± 4 + = ( ± ) 5 T [] = gives α + β =. T [] = gives ( ) ( ) + 5 5 α + β = α β = 5 6.3 The Characteristic Polynomial Ernst Mayr, Harald Räcke 83 6.3 The Characteristic Polynomial Ernst Mayr, Harald Räcke 84
Example: Fibonacci Sequence The Inhomogeneous Case Consider the recurrence relation: Hence, the solution is [( ) n + 5 5 ( ) n ] 5 c T (n) + c T (n ) + c T (n ) + + c k T (n k) = f (n) with f (n). While we have a fairly general technique for solving homogeneous, linear recurrence relations the inhomogeneous case is different. 6.3 The Characteristic Polynomial Ernst Mayr, Harald Räcke 85 6.3 The Characteristic Polynomial Ernst Mayr, Harald Räcke 86 The Inhomogeneous Case The Inhomogeneous Case Example: T [n] = T [n ] + T [] = The general solution of the recurrence relation is T (n) = T h (n) + T p (n), where T h is any solution to the homogeneous equation, and T p is one particular solution to the inhomogeneous equation. Then, T [n ] = T [n ] + (n ) Subtracting the first from the second equation gives, T [n] T [n ] = T [n ] T [n ] (n ) There is no general method to find a particular solution. or T [n] = T [n ] T [n ] (n ) I get a completely determined recurrence if I add T [] = and T [] =. 6.3 The Characteristic Polynomial Ernst Mayr, Harald Räcke 87 6.3 The Characteristic Polynomial Ernst Mayr, Harald Räcke 88
The Inhomogeneous Case Example: Characteristic polynomial: The Inhomogeneous Case If f (n) is a polynomial of degree r this method can be applied r + times to obtain a homogeneous equation: λ } {{ λ + } = (λ ) Shift: T [n] = T [n ] + n Then the solution is of the form T [n] = α n + βn n = α + βn T [] = gives α =. T [] = gives + β = β =. T [n ] = T [n ] + (n ) = T [n ] + n n + Difference: T [n] T [n ] = T [n ] T [n ] + n T [n] = T [n ] T [n ] + n 6.3 The Characteristic Polynomial Ernst Mayr, Harald Räcke 89 6.3 The Characteristic Polynomial Ernst Mayr, Harald Räcke 9 T [n] = T [n ] T [n ] + n 6.4 Generating Functions Shift: T [n ] = T [n ] T [n 3] + (n ) = T [n ] T [n 3] + n 3 Definition 7 (Generating Function) Let (a n ) n be a sequence. The corresponding generating function (Erzeugendenfunktion) is Difference: F(z) := n a n z n ; T [n] T [n ] =T [n ] T [n ] + n T [n ] + T [n 3] n + 3 T [n] = 3T [n ] 3T [n ] + T [n 3] + exponential generating function (exponentielle Erzeugendenfunktion) is F(z) = n a n n! zn. and so on... 6.3 The Characteristic Polynomial Ernst Mayr, Harald Räcke 9 6.4 Generating Functions Ernst Mayr, Harald Räcke 9
6.4 Generating Functions 6.4 Generating Functions There are two different views: Example 8. The generating function of the sequence (,,,...) is F(z) =.. The generating function of the sequence (,,,...) is F(z) = z. A generating function is a formal power series (formale Potenzreihe). Then the generating function is an algebraic object. Let f = n a n z n and g = n b n z n. Equality: f and g are equal if a n = b n for all n. Addition: f + g := n (a n + b n )z n. Multiplication: f g := n c n z n with c = n p= a p b n p. There are no convergence issues here. 6.4 Generating Functions Ernst Mayr, Harald Räcke 93 6.4 Generating Functions Ernst Mayr, Harald Räcke 94 6.4 Generating Functions 6.4 Generating Functions The arithmetic view: We view a power series as a function f : C C. Then, it is important to think about convergence/convergence radius etc. What does n z n = z mean in the algebraic view? It means that the power series z and the power series n z n are invers, i.e., ( ) ( z n z n) =. This is well-defined. 6.4 Generating Functions Ernst Mayr, Harald Räcke 95 6.4 Generating Functions Ernst Mayr, Harald Räcke 96
6.4 Generating Functions Suppose we are given the generating function n We can compute the derivative: z n = z. nz n = n }{{} n (n+)z n ( z) Formally the derivative of a formal power series n a n z n is defined as n na n z n. The known rules for differentiation work for this definition. In particular, e.g. the derivative of z is ( z). Note that this requires a proof if we consider power series as algebraic objects. However, we did not prove this in the lecture. Hence, the generating function of the sequence a n = n + is /( z). 6.4 Generating Functions We can repeat this Derivative: (n + )z n = ( z). n n(n + )z n = n n (n+)(n+)z n }{{} ( z) 3 Hence, the generating function of the sequence a n = (n + )(n + ) is ( z) 3. 6.4 Generating Functions Ernst Mayr, Harald Räcke 97 6.4 Generating Functions Ernst Mayr, Harald Räcke 98 6.4 Generating Functions 6.4 Generating Functions Computing the k-th derivative of z n. n(n )... (n k + )z n k = n k n k! = ( z) k+. Hence: ( ) n + k z n = k ( z) k+. n (n + k)... (n + )z n nz n = + )z n n (n n n = ( z) z z = ( z) The generating function of the sequence a n = n is z n z ( z). The generating function of the sequence a n = ( ) n+k k is ( z) k+. 6.4 Generating Functions Ernst Mayr, Harald Räcke 99 6.4 Generating Functions Ernst Mayr, Harald Räcke
6.4 Generating Functions We know Hence, n n y n = y a n z n = az The generating function of the sequence f n = a n is az. Example: a n = a n +, a = Suppose we have the recurrence a n = a n + for n and a =. A(z) = a n z n n = a + (a n + )z n n = + z a n z n + n n = z a n z n + z n n n = za(z) + n z n z n = za(z) + z 6.4 Generating Functions Ernst Mayr, Harald Räcke 6.4 Generating Functions Ernst Mayr, Harald Räcke Example: a n = a n +, a = Some Generating Functions Solving for A(z) gives a n z n = A(z) = ( z) n = (n + )z n n Hence, a n = n +. n-th sequence element n + ( n+k k n generating function z ( z) ) ( z) k+ z ( z) a n az n z( + z) ( z) 3 n! e z 6.4 Generating Functions Ernst Mayr, Harald Räcke 3 6.4 Generating Functions Ernst Mayr, Harald Räcke 4
Some Generating Functions Solving Recursions with Generating Functions n-th sequence element cf n f n + g n n i= f ig n i generating function cf F + G F G f n k (n k); otw. z k F n i= f i nf n c n f n F(z) z z df(z) dz F(cz). Set A(z) = n a n z n.. Transform the right hand side so that boundary condition and recurrence relation can be plugged in. 3. Do further transformations so that the infinite sums on the right hand side can be replaced by A(z). 4. Solving for A(z) gives an equation of the form A(z) = f (z), where hopefully f (z) is a simple function. 5. Write f (z) as a formal power series. Techniques: partial fraction decomposition (Partialbruchzerlegung) lookup in tables 6. The coefficients of the resulting power series are the a n. 6.4 Generating Functions Ernst Mayr, Harald Räcke 5 6.4 Generating Functions Ernst Mayr, Harald Räcke 6 Example: a n = a n, a =. Set up generating function: A(z) = a n z n n Example: a n = a n, a = 3. Transform right hand side so that infinite sums can be replaced by A(z) or by simple function. A(z) = + (a n )z n n. Transform right hand side so that recurrence can be plugged in: A(z) = a + a n z n n = + z a n z n n = + z a n z n n. Plug in: A(z) = + (a n )z n n 4. Solve for A(z). = + z A(z) A(z) = z 6.4 Generating Functions Ernst Mayr, Harald Räcke 7 6.4 Generating Functions Ernst Mayr, Harald Räcke 8
Example: a n = a n, a = 5. Rewrite f (z) as a power series: a n z n = A(z) = z = n z n n n Example: a n = 3a n + n, a =. Set up generating function: A(z) = a n z n n 6.4 Generating Functions Ernst Mayr, Harald Räcke 9 6.4 Generating Functions Ernst Mayr, Harald Räcke Example: a n = 3a n + n, a =./3. Transform right hand side: A(z) = a n z n n Example: a n = 3a n + n, a = 4. Solve for A(z): A(z) = + 3zA(z) + z ( z) = a + a n z n n = + (3a n + n)z n n = + 3z a n z n + nz n n n = + 3z a n z n + nz n n n z = + 3zA(z) + ( z) gives A(z) = ( z) + z ( 3z)( z) = z z + ( 3z)( z) 6.4 Generating Functions Ernst Mayr, Harald Räcke 6.4 Generating Functions Ernst Mayr, Harald Räcke
Example: a n = 3a n + n, a = 5. Write f (z) as a formal power series: We use partial fraction decomposition: z z + ( 3z)( z)! = A 3z + B z + C ( z) Example: a n = 3a n + n, a = 5. Write f (z) as a formal power series: This leads to the following conditions: A + B + C = A + 4B + 3C = This gives z z + = A( z) + B( 3z)( z) + C( 3z) = A( z + z ) + B( 4z + 3z ) + C( 3z) = (A + 3B)z + ( A 4B 3C)z + (A + B + C) which gives A + 3B = A = 7 4 B = 4 C = 6.4 Generating Functions Ernst Mayr, Harald Räcke 3 6.4 Generating Functions Ernst Mayr, Harald Räcke 4 Example: a n = 3a n + n, a = 6.5 Transformation of the Recurrence 5. Write f (z) as a formal power series: A(z) = 7 4 3z 4 z ( z) = 7 4 3 n z n 4 z n (n + )z n n n = ( 7 4 3n 4 ) (n + ) z n n = ( 7 4 3n n 3 z 4) n n 6. This means a n = 7 4 3n n 3 4. n Example 9 Define Then f = f = f n = f n f n for n. g n := log f n. g n = g n + g n for n g = log = (for log = log ), g = g n = F n (n-th Fibonacci number) f n = F n 6.4 Generating Functions Ernst Mayr, Harald Räcke 5 6.5 Transformation of the Recurrence Ernst Mayr, Harald Räcke 6
6.5 Transformation of the Recurrence 6 Recurrences Example f = f n = 3f n + n; for n = k, k ; We get g k = 3 [ ] g k + k [ = 3 3g k + k ] + k = 3 [ g k ] + 3 k + k Define Then: g = g k := f k. g k = 3g k + k, k = 3 [ 3g k 3 + k ] + 3 k + k = 3 3 g k 3 + 3 k + 3 k + k = k k ( 3 i ) i= = k ( 3 )k+ / = 3 k+ k+ 6.5 Transformation of the Recurrence Ernst Mayr, Harald Räcke 7 6.5 Transformation of the Recurrence Ernst Mayr, Harald Räcke 8 6 Recurrences 6 Recurrences Let n = k : g k = 3 k+ k+, hence f n = 3 3 k k = 3( log 3 ) k k = 3( k ) log 3 k = 3n log 3 n. Bibliography [MS8] [CLRS9] [Liu85] Kurt Mehlhorn, Peter Sanders: Algorithms and Data Structures The Basic Toolbox, Springer, 8 Thomas H. Cormen, Charles E. Leiserson, Ron L. Rivest, Clifford Stein: Introduction to algorithms (3rd ed.), MIT Press and McGraw-Hill, 9 Chung Laung Liu: Elements of Discrete Mathematics McGraw-Hill, 985 The Karatsuba method can be found in [MS8] Chapter. Chapter 4.3 of [CLRS9] covers the Substitution method which roughly corresponds to Guessing+induction. Chapters 4.4, 4.5, 4.6 of this book cover the master theorem. Methods using the characteristic polynomial and generating functions can be found in [Liu85] Chapter. 6.5 Transformation of the Recurrence Ernst Mayr, Harald Räcke 9 6.5 Transformation of the Recurrence Ernst Mayr, Harald Räcke
Abstract Data Type Part III Data Structures An abstract data type (ADT) is defined by an interface of operations or methods that can be performed and that have a defined behavior. The data types in this lecture all operate on objects that are represented by a [key, value] pair. The key comes from a totally ordered set, and we assume that there is an efficient comparison function. The value can be anything; it usually carries satellite information important for the application that uses the ADT. Ernst Mayr, Harald Räcke Ernst Mayr, Harald Räcke Dynamic Set Operations Dynamic Set Operations S. search(k): Returns pointer to object x from S with key[x] = k or null. S. insert(x): Inserts object x into set S. key[x] must not currently exist in the data-structure. S. delete(x): Given pointer to object x from S, delete x from the set. S. minimum(): Return pointer to object with smallest key-value in S. S. maximum(): Return pointer to object with largest key-value in S. S. successor(x): Return pointer to the next larger element in S or null if x is maximum. S. predecessor(x): Return pointer to the next smaller element in S or null if x is minimum. S. union(s ): Sets S := S S. The set S is destroyed. S. merge(s ): Sets S := S S. Requires S S =. S. split(k, S ): S := {x S key[x] k}, S := {x S key[x] > k}. S. concatenate(s ): S := S S. Requires S. maximum() S. minimum(). S. decrease-key(x, k): Replace key[x] by k key[x]. Ernst Mayr, Harald Räcke Ernst Mayr, Harald Räcke 3
Examples of ADTs 7 Dictionary Stack: S.push(x): Insert an element. S.pop(): Return the element from S that was inserted most recently; delete it from S. S.empty(): Tell if S contains any object. Queue: S.enqueue(x): Insert an element. S.dequeue(): Return the element that is longest in the structure; delete it from S. Dictionary: S.insert(x): Insert an element x. S.delete(x): Delete the element pointed to by x. S.search(k): Return a pointer to an element e with key[e] = k in S if it exists; otherwise return null. S.empty(): Tell if S contains any object. Priority-Queue: S.insert(x): Insert an element. S.delete-min(): Return the element with lowest key-value; delete it from S. 7 Dictionary Ernst Mayr, Harald Räcke 5 7. Binary Search Trees An (internal) binary search tree stores the elements in a binary tree. Each tree-node corresponds to an element. All elements in the left sub-tree of a node v have a smaller key-value than key[v] and elements in the right sub-tree have a larger-key value. We assume that all key-values are different. (External Search Trees store objects only at leaf-vertices) Examples: 7. Binary Search Trees We consider the following operations on binary search trees. Note that this is a super-set of the dictionary-operations. T. insert(x) T. delete(x) T. search(k) T. successor(x) 6 T. predecessor(x) 7 T. minimum() 5 8 5 6 T. maximum() 7 8 7. Binary Search Trees Ernst Mayr, Harald Räcke 6 7. Binary Search Trees Ernst Mayr, Harald Räcke 7
Binary Search Trees: Searching TreeSearch(root, 7) 5 Binary Search Trees: Searching TreeSearch(root, 8) 5 3 3 3 3 6 6 48 6 6 48 3 9 6 3 9 43 5 3 9 6 3 9 43 5 5 7 4 9 4 8 4 47 55 5 7 4 9 4 8 4 47 55 4 7 4 7 Algorithm 5 TreeSearch(x, k) : if x = null or k = key[x] return x : if k < key[x] return TreeSearch(left[x], k) 3: else return TreeSearch(right[x], k) Algorithm 5 TreeSearch(x, k) : if x = null or k = key[x] return x : if k < key[x] return TreeSearch(left[x], k) 3: else return TreeSearch(right[x], k) 7. Binary Search Trees Ernst Mayr, Harald Räcke 8 7. Binary Search Trees Ernst Mayr, Harald Räcke 9 Binary Search Trees: Minimum Binary Search Trees: Successor 5 3 3 5 3 3 succ is min in right sub-tree 6 6 48 6 6 48 3 9 6 3 9 43 5 3 9 6 3 9 43 5 5 7 4 9 4 8 4 47 55 5 7 4 9 8 47 55 4 7 4 7 Algorithm 7 TreeSucc(x) Algorithm 6 TreeMin(x) : if x = null or left[x] = null return x : return TreeMin(left[x]) : if right[x] null return TreeMin(right[x]) : y parent[x] 3: while y null and x = right[y] do 4: x y; y parent[x] 5: return y; 7. Binary Search Trees Ernst Mayr, Harald Räcke 3 7. Binary Search Trees Ernst Mayr, Harald Räcke 3
Binary Search Trees: Successor 5 y Binary Search Trees: Insert Insert element not in the tree. TreeInsert(root, ) 5 x 3 3 3 3 3 5 4 6 9 7 6 4 9 7 succ is lowest ancestor going left to reach me 3 7. Binary Search Trees Ernst Mayr, Harald Räcke 3 6 8 9 Algorithm 7 TreeSucc(x) : if right[x] null return TreeMin(right[x]) : y parent[x] 3: while y null and x = right[y] do 4: x y; y parent[x] 5: return y; 43 47 48 5 55 3 5 4 6 9 7 6 4 9 7 Search for z. At some point the search stops at a null-pointer. This is the place to insert z. 3 6 Algorithm 8 TreeInsert(x, z) : if x = null then 9 : root[t ] z; parent[z] null; 3: return; 8 47 55 4: if key[x] > key[z] then 5: if left[x] = null then 6: left[x] z; parent[z] x; 7: else TreeInsert(left[x], z); 8: else 9: if right[x] = null then : right[x] z; parent[z] x; : else TreeInsert(right[x], z); 43 48 5 Binary Search Trees: Delete 5 Binary Search Trees: Delete 5 3 3 3 3 6 6 48 6 6 48 3 9 6 3 9 43 5 3 9 6 3 9 43 5 5 7 4 9 4 8 4 47 55 5 7 4 9 4 8 4 47 55 4 7 4 4 7 4 Case : Element does not have any children Simply go to the parent and set the corresponding pointer to null. Case : Element has exactly one child Splice the element out of the tree by connecting its parent to its successor.
Binary Search Trees: Delete Binary Search Trees: Delete 5 3 5 4 6 9 7 3 3 6 4 9 7 Case 3: Element has two children 3 4 Find the successor of the element 9 48 43 4 47 4 5 55 Algorithm 9 TreeDelete(z) : if left[z] = null or right[z] = null : then y z else y TreeSucc(z); 3: if left[y] null 4: then x left[y] else x right[y]; 5: if x null then parent[x] parent[y]; 6: if parent[y] = null then 7: root[t ] x 8: else 9: if y = left[parent[y]] then : left[parent[y]] x : else : right[parent[y]] x 3: if y z then copy y-data to z select y to splice out x is child of y (or null) parent[x] is correct fix pointer to x fix pointer to x Splice successor out of the tree Replace content of element by content of successor 7. Binary Search Trees Ernst Mayr, Harald Räcke 34 Balanced Binary Search Trees Binary Search Trees (BSTs) All operations on a binary search tree can be performed in time O(h), where h denotes the height of the tree. However the height of the tree may become as large as Θ(n). Balanced Binary Search Trees With each insert- and delete-operation perform local adjustments to guarantee a height of O(log n). AVL-trees, Red-black trees, Scapegoat trees, -3 trees, B-trees, AA trees, Treaps Bibliography [MS8] [CLRS9] Kurt Mehlhorn, Peter Sanders: Algorithms and Data Structures The Basic Toolbox, Springer, 8 Thomas H. Cormen, Charles E. Leiserson, Ron L. Rivest, Clifford Stein: Introduction to Algorithms (3rd ed.), MIT Press and McGraw-Hill, 9 Binary search trees can be found in every standard text book. For example Chapter 7. in [MS8] and Chapter in [CLRS9]. similar: SPLAY trees. 7. Binary Search Trees Ernst Mayr, Harald Räcke 35 7. Binary Search Trees Ernst Mayr, Harald Räcke 36