MA008 p.1/37 MA008/MIIZ01 Design and Analysis of Algorithms Lecture Notes 3 Dr. Markus Hagenbuchner markus@uow.edu.au.
MA008 p.2/37 Exercise 1 (from LN 2) Asymptotic Notation When constants appear in exponents or as the base of an exponential, they are significant. Thus, n 2 O(n 3 ) and 2 n O(3 n ) (Can you prove it?).
MA008 p.3/37 Exercise 1: Solution Rules needed: Asymptotic Notation Asymptotic Form Relationship Definition f(n) Θ(g(n)) f(n) g(n) 0 < lim n f(n) g(n) <. f(n) O(g(n)) f(n) g(n) 0 lim n f(n) g(n) <. f(n) Ω(g(n)) f(n) g(n) 0 < lim n f(n) g(n). Case 1: n 2 O(n 3 ) Proof: lim n n 2 n 3 = lim n Case 2: 2 n O(3 n ) n n n n n = lim n 1 n = 0 2 Proof: lim n 2 2... 2 n 3 = lim n n 3 3... 3 = lim n 2 n 3 = 0
MA008 p.4/37 Exercise 2 Prove property 5: (log b x = log b a log a x).
MA008 p.4/37 Exercise 2 Prove property 5: (log b x = log b a log a x). Solution through substitution: Denote log b x = i, log b a = j, and log a x = k. Then b i = x,b j = a and a k = x. We have b i = x = a k = (b j ) k = b jk. Hence, i = jk, and the result follows.
How to Write Algorithms MA008 p.5/37
MA008 p.6/37 How to Write Algorithms When writing pseudo-code, do NOT write a complete program with variable and declarations. Explain the algorithm intuitively in English, and then give a high-level pseudocode description. The level of detail should be just enough that a competent programmer could take your explanation and implement it without undue ambiguity. You may assume that simple data structures have been provided for you. So, it is clearer to say Append x to the end of list L, rather than giving low-level code and pointer manipulation.
MA008 p.7/37 How to Write Algorithms It is often a good idea to provide an example of your algorithm s recurtion on a concrete example. Reading code is hard, but a good figure makes things much earlier to understand. It also provides the grader with more understanding of your intentions this can mean partial credit in case you make a coding error. Providing a figure is always a good idea. Insert comments to inform the reader of logical relations that hold at various places in your algorithm.
MA008 p.8/37 Exercise: Sorting Sorting is among the most basic computational problems in algorithm design. Sorting is to permute the items in an array so that they are arranged in increasing order. Sorting is important because it is often used in more complex algorithms to allow faster subsequent retrieval (ie. binary search). Task 1: Define a brute-force algorithm which sorts an array of elements in an array. Task 2: Analyse the computational time complexity of the brute-force sorting algorithm.
Sorting algorithms Important sorting algorithms are mergesort, quicksort, heapsort, bubblesort. Good sorting algorithms can sort an array in Θ(n log n). But how? We distinguish between two properties: Inplace: The algorithm uses no additional array storages, and hence it is possible to sort very large lists without the need to allocate additional arrays. Stable: A sorting algorithm is stable if two elements that are equal remain in the same relative position after sorting is completed. MA008 p.9/37
Divide-and-Conquer Algorithms MA008 p.10/37
MA008 p.11/37 Introduction An important approach to algorithm design is based on divide-and-conquer. The general strategy of divide-and-conquer is to divide a problem instance into one (or more) subproblems of the same type as the original problem. These subproblem are then solved separately (by the same method) and then their solutions are combined in some fashion to give the solution to the original problem instance. An implementation of a divide-and-conquer algorithm is often recursive.
MA008 p.12/37 Binary search Search a sorted array for a particular value called the key. First check the middle element of the array. If it equals key, then we re done; If it is less than key, then we do a binary search of the upper half of the array; if it is greater than key, then we do a binary search of the lower half of the array. (What is the expected complexity of this algorithm?)
MA008 p.13/37 Max-min problem We want to find the maximum and minimum values in an unsorted array. We first break the array in half, and find the max and min of each half separately (i.e. we solve both subproblems). Then, the min is the smallest of the two minimums, and the max is the larger of the two maximums. (What is the expected complexity of this algorithm?)
MA008 p.14/37 Divide-and-Conquer Algorithms Divide-and-conquer algorithms consist of three basic steps: divide, conquer and combine. Divide: Split the original problem of size n into a (typically 2, but maybe more) problems of roughly equal sizes, say n/b. Conquer: Solve each subproblem recursively. Combine: Combine the solutions to the subproblem to a solution to the original problem. The time to combine the solutions is called the overhead. We will assume that the running time of the overhead is some polynomial of n, say cn k, for constants c and k.
MA008 p.15/37 Mergesort This recursive subdivision is repeated until the size of the subproblems is small enough that the problem can be solved by brute-force. Mergesort is a practical divide-and-conquer algorithm. It sorts a list of numbers by first splitting the list into two sublists of roughly equal size, sort each sublist, and then merge the sorted lists into a single sorted list. we subdivide each problem into a = 2 parts, each part of size n/2. two sorted lists of size n/2 can be merged into a single sorted list of size n in Θ(n) time (thus c = k = 1).
MA008 p.16/37 Mergesort 12 23 3 13 8 4 11 24 Split 12 23 3 13 8 4 11 24 Sort each sublist 3 12 13 23 4 8 11 24 Merge 3 4 8 11 12 13 23 24
MA008 p.17/37 Analysis How long do divide-and-conquer algorithms take to run? Let T(n) be the function that describes the running time of the algorithm on a subarray of length n 1. As a basis, when n is 1, the algorithm runs in constant time, namely Θ(1) T(1) = 1. If n > 1, split into two sublists, each size n/2. Make two calls on these arrays, each taking T(n/2) time. Merging is done in linear time (expressed as Θ(n)), can you see why?
MA008 p.18/37 Analysis So the overall running time is described by the following recurrence, which is defined for all n 1, when n is a power of 2: T(n) = { 1 if n = 1, 2T(n/2) + n otherwise. Recurrence: An equation or inequality describing a function in terms of its values on smaller inputs. Need method(s) to solve recurences in order to obtain asymptotic bounds.
MA008 p.19/37 The Master Theorem Is the easiest method to solving recurrences. Master Theorem: Let a 1,b 1 be constants and let T(n) be the recurrence T(1) = 1, T(n) = a T(n/b) + cn k (n > 1). Then the growth rate of T(n) is: T(n) = Θ(n log b a ), if a > b k Θ(n k log n), if a = b k Θ(n k ), if a < b k. Q: Asymptotic bound of mergesort?
Master Theorem We observe that the growth rates of T(n) depends on a,b, and k, but not on c. (why?) From the Master Theorem, we see that in our recurrence for Mergesort a = 2,b = 2, and k = 1, so a = b k. Thus T(n) is Θ(n log n). There are many recurrences that cannot be put into this form. For example, the following recurrence is quite common: T(n) = 2T(n/2) + n log n. This solves to T(n) Θ(n log 2 n), but not from the Master Theorem. For such recurrence, other methods are needed. MA008 p.20/37
MA008 p.21/37 Iteration A more basic method for solving recurrences is that of iteration( or expansion). This is a rather painstaking process of repeatedly applying the definition of the recurrence until (hopefully) a simple pattern emerges. This pattern usually results in a summation that is easy to solve. The (lengthy) proof of the Master Theorem is based on iteration.
MA008 p.22/37 Example Let us consider applying this to the following recurrence. We assume that n is a power of 3. T(1) = 1 T(n) = 2T(n/3) + n if n > 1. First we expand the recurrence into a summation, until seeing the general pattern emerge.
Iteration example T(n) = 2T( n 3 ) + n = 2(2T( n 9 ) + n 3 ) + n = 4T( n 9 ) + (n + 2n 3 ) = 4(2T( n 27 ) + n 9 ) + (n + 2n 3 ) = 8T( n 27 ) + (n + 2n 3 + 4n 9 ) = 2 k T( n k 1 3 k ) + i=0 2 i n 3 i = 2 k T( n k 1 3 k ) + n (2/3) i. MA008 p.23/37
Iteration example k is the number of iterations. To know how many expansions are needed we need to arrive at the basis case. To do this we set n/(3 k ) = 1, meaning that k = log 3 n. Substituting this, and using the identity a log b = b log a (rule 6, LN2), we obtain T(n) = 2 log 3 n T(1) + n log 3 n 1 i=0 (2/3) i = n log 3 3 + n log 3 n 1 i=0 (2/3) i. (Who can spot an error in the above?) MA008 p.24/37
MA008 p.25/37 Iteration example Next, apply the geometric series to get: T(n) = n log 3 2 + n 1 (2/3)log 3 n 1 (2/3) = n log 3 2 + 3n(1 (2/3) log 3 n ) = n log 3 2 + 3n(1 n log 3(2/3) ) = n log 3 2 + 3n(1 n (log 3 2) 1 ) = n log 3 2 + 3n 3n log 3 2 = 3n 2n log 3 2. Since log 3 2 0.631 < 1,T(n) is dominated by the 3n term asymptotically, and so it is Θ(n).
MA008 p.26/37 Exercise 1 1. Design a brute-force algorithm for computing the value of a polynomial p(x) = a n x n + a n 1 x n 1 + + a 1 x + a 0 at a given point x, and determine its worst case efficiency class. 2. If the algorithm you designed is in Θ(n 2 ), design a linear efficiency algorithm for this problem. 3. Is it possible to design an algorithm with a better than linear efficiency for this problem?
MA008 p.27/37 Exercise 2 On a sheet of paper, show each step of sorting the list E, X, A, M, P, L, E in alphabetical order by mergesort. Answer the following questions: Is mergesort an inplace sorting algorithm? Is mergesort a stable sorting algorithm?
Induction and Constructive Induction Another technique for solving recurrences (and this works for summations as well) is to guess the solution, or the general form of the solution, then attempt to verify its correctness through induction. Sometimes there are parameters whose values you do not know. This is fine. In the course of the induction proof, you will usually find out what this values must be. We will consider a famous example, that of the Fibonacci numbers. F 0 = 0 F 1 = 1 F n = F n 1 + F n 2 for n 2. MA008 p.28/37
The Fibonacci numbers... arise in data structure design. If you study Adelson-Velskii and Landis (AVL, height balanced tree) in Data structure and Algorithm], you have learnt that minimum AVL trees are produced by the recursive construction given below. Let L(i) denote the number of leaves in the minimum-sized AVL tree of height i. To construct a minimum-sized AVL tree of height i, you create a root node whose children consist of a minimum-sized AVL tree of heights i 1 and i 2. Thus the number of leaves obeys L(0) = L(1) = 1,L(i) = (L(i 1) + L(i 2). It is easy to see that L(i) = F i+1. MA008 p.29/37
MA008 p.30/37 L(0) = 1 L(1) = 1 L(2) = 2 L(3) = 3 L(4) = 5 Minimum-sized AVL trees
MA008 p.31/37 If you expand the Fibonacci series for a number of terms, you will observe that F n appears to grow exponentially, but not as fast as 2 n. It is tempting to conjecture that F n φ n 1 for some real parameter φ, where 1 φ 2. We can use induction to prove this and derive a bound on φ. Lemma: For all integers n 1,F n φ n 1 for some constant φ, 1 < φ < 2.
MA008 p.32/37 Proof We will try to derive the tightest bound we can on the value φ. Basic: For the basis cases we consider n = 1. Observe that F 1 = 1 φ 0, as desired. Induction step: For the induction step, let us assume that F m φ m 1 whenever 1 m < n. Using this induction hypothesis we will show that the lemma holds for n itself, whenever n 2.
MA008 p.33/37 Proof (cont.) Since n 2, we have F n = F n 1 + F n 2. Now, since n 1 and n 2 are both strictly less than n, we can apply the induction hypothesis, from which we have F n φ n 2 + φ n 3 = φ n 3 (1 + φ). we want to show that this is at most φ n 1 (for a suitable choice of φ). Clearly this will be true if and only if (1 + φ) φ 2. This is not true for all values of φ (for example it is not true when φ = 1 but it is true when φ = 2.
MA008 p.34/37 Proof (cont.) At the critical value of φ this inequality will be an equality, implying that we want to find the roots of the equation φ 2 φ 1 = 0. By the quadratic formula we have φ = 1 ± 1 + 4 2 = 1 ± 5. 2 Since 5 2.24, observe that one of roots is negative, and hence would not be a possible candidate for φ. The positive root is φ = 1 + 5 2 1.618.
MA008 p.35/37 There is a very subtle bug in the preceding proof. Can you spot it? The error occurs in the case n = 2. Here we claim that F 2 = F 1 + F 0 and then apply the induction hypothesis to both F 1 and F 0. But the induction hypothesis only applies for m 1, and hence cannot be applied to F 0 To fix it we could include F 2 as part of the basis case as well.
Notice not only did we prove the lemma by induction, but actually determined the value of φ which makes the lemma true. This is why this method is called constructive induction. The value φ = 1 2 (1 + 5) is a famous constant in mathematics, architecture and art. It is the golden ratio. Two numbers A and B satisfy the golden ration if A B = A + B A. It is easy to verify that A = φ and B = 1 satisfies this condition. This proportion occurs throughout the world of art and architecture. MA008 p.36/37
MA008 p.37/37 Next lecture More on divide-and-conquer Greedy algorithms