Design and Analysis of Algorithms

Transcription

1 CSE 101, Winter 2018 Design and Analysis of Algorithms Lecture 4: Divide and Conquer (I) Class URL:

2 Divide and Conquer ( DQ ) First paradigm or framework DQ(S) if S is small return ADHOC(S) else decompose S into subproblems S 1,, S k // divide for i = 1 to k do y i = DQ(S i ) // conquer recombine y i into solution y // compose return y Universal method : Mergesort, Quicksort, FFT, Matrix/Integer arithmetic are classic examples

3 DQ for Sorting From CSE 21 Divide (into two equal parts) Conquer (solve for each part separately) Combine separate solutions Mergesort Divide into two equal parts Sort each part using Mergesort (recursion!!!) Merge two sorted subsequences (Note: Quicksort is another DQ algorithm for sorting)

4 Merging Two Subsequences From CSE 21 Sequence #1: x[1], x[2],, x[m] Sequence #2: y[1], y[2],, y[n] if y[i] < x[j] then compare y[i+1] and x[j], else compare y[i] and x[j+1] After comparing x[1] and y[1], we advance in x[.] and y[.] a total of (m-1) + (n-1) times m + n - 1 comparisons (edges between red and blue) = linear time

5 Mergesort Tree of Recursive Subproblems DIVIDE log n log n CONQUER, COMBINE (n comparisons per level) X (log n levels) = (n log n runtime)

6 Integer Multiplication From CSE 21 Multiplying Large Integers (Karatsuba-Ofman) A = r n-1 a n r 1 a 1 + a 0, r = radix Classic approach = what you learned 12 years ago a n-1 a n-2 a 1 a 0 X b n-1 b n-2 b 1 b (n 2 ) work

7 (Unhelpful) DQ for Integer Multiplication Can we apply D/Q? Let n = 2s, r = 10 radix A = wx, B = yz AB = xz + 10 s (wz + xy) s wy Complexity analysis 4 n/2-digit multiplications: xz, wz, xy, wy Digit-shifting: multiplication by 10 s, 10 2s 3 additions T(n) = 4T(n/2) + (n) T(n) 4 T(n/2) + cn 4 [ 4T(n/4) + cn/2] + cn = 16 T(n/4) + (1 + 2)cn 16 [4T(n/8) + cn/4] + (1 + 2)cn = 64 T(n/8) + ( )cn 4 k T(n/2 k ) + ( k-1 )cn For k = log 2 n: T(n) n 2 T(1) + cn 2 = O(n 2 )

8 From CSE 21 Multiplication: HOW Divide and conquer? Divide n-digit numbers into two n/2-digit numbers. If a = and b = , we can write To multiply: a = (1234) * (5678) b = (2468) * (1357) ((1234) * (5678))((2468) * (1357))= (1234)(2468) * (1234)(1357) * (2468)(5678) * (1357)(5678)

9 From CSE 21 One 8-digit multiplication Multiplication: WHEN ( )( )=((1234) * (5678))((2468) * (1357))= (1234)(2468) * (1234)(1357) * (2468)(5678) * (1357)(5678) Four 4-digit multiplications (plus some shifts, sums)

10 (Helpful) DQ for Integer Multiplication Observation: r = (w+x)(y+z) = wy + (wz+xy) + xz r (w+x)(y+z) p wy q xz return 10 2s p + 10 s (r -p-q) + q What is (x n 1) / (x 1)? T(n) = 3T(n/2) + (n) T(n) 3 T(n/2) + cn 3 [ 3T(n/4) + cn/2] + cn = 9 T(n/4) + (1 + 3/2)cn 9 [3T(n/8) + cn/4] + (1 + 3/2)cn = 27 T(n/8) + (1 + 3/2 + 9/4)cn 3 k T(n/2 k ) + (1 + 3/2 + 9/4 + + (3/2) k-1 )cn For k = log 2 n: T(n) 3 log 2n T(1) + cn 2 (3/2) log 2n = O(3 log 2n ) = O(n log 23 ) = O(n ) [(3/2) k 1] / (3/2 1) = 2[(3/2) k 1] a log bc = c log ba (Take log base b)

11 Karatsuba Multiplication: HOW From CSE 21 Rosen p. 528 ( )( )=((1234) * (5678))((2468) * (1357))= (1234)(2468) * (1234)(1357) * (2468)(5678) * (1357)(5678) (1234)(2468) * ( ) + [(1234) - (5678)][ (1357)-(2468) ]* (1357)(5678) * ( ) Insight: replace one (of the 4) multiplications by (linear time) subtraction

12 Karatsuba Multiplication: WHEN From CSE 21 Instead of Rosen p. 528 T(n) = 4 T(n/2) + cn T K (n) = 3 T K (n/2) + pn with c a constant, now we have with p a constant.

13 Matrix Multiplication A = [ ], B = [ ] are n x n matrices a 11, a 12, etc are n/2 x n/2 submatrices M = AB = [ ] where m 11 = a 11 b 11 + a 12 b 21 etc. Evaluation requires 8 multiplies, 4 adds T(n) = 8T(n/2) + O(n 2 ) (n 3 ) (will see why later) a 11 a 12 a 21 a 22 b 11 b 12 X = b 21 b 22 m 11 m 12 m 21 m 22

14 Strassen s Matrix Multiplication p 1 = (a 21 + a 22 -a 11 )(b 22 -b 12 + b 11 ) p 2 = a 11 b 11 p 3 = a 12 b 21 p 4 = (a 11 -a 21 )(b 22 -b 12 ) p 5 = (a 21 +a 22 )(b 12 -b 11 ) p 6 = (a 12 -a 21 +a 11 -a 22 )b 22 p 7 = a 22 (b 11 +b 22 -b 12 -b 21 ) AB 11 = p 2 + p 3 AB 12 = p 1 + p 2 + p 5 + p 6 AB 21 = p 1 + p 2 + p 4 + p 7 AB 22 = p 1 + p 2 + p 4 + p 5 T(n) = 7T(n/2) + O(n) // 7 multiplies, 24 adds Can get to 15 adds (n 2.81 ) (will see why later)

15 General Formula (aka Master Theorem ) Recurrence T(n) a T(n/b) + O(n d ) From CSE 21 1) T(n) = O(n d ) if a < b d 2) T(n) = O(n d log n) if a = b d 3) T(n) = O(n log ba ) if a > b d Type (3): integer multiplication, Strassen Type (2): mergesort

16 Master Theorem Examples Mergesort T(n) = 2T(n/2) + (n) T ( n) ( n log n) Strassen Matrix Multiply T(n) = 7T(n/2) + (n 2 ) T ( n) 7 ( n log 2 ) ( n )

17 Recurrence: T(n) a T(n/b) + O(n d ) Anatomy of the Master Theorem

18 Anatomy of the Master Theorem T(n) a T(n/b) + O(n d ) Understand this picture and the notation conventions! n is the ( original, parent ) instance size a is the branching factor (i.e., number of subproblems created from a given problem) This is inherent to the recurrence for T(n) b is the size decrease factor E.g., b = 2 when subproblems are half the size of their parent problem d is the exponent in the + O(n d ) term of the recurrence

19 Anatomy of the Master Theorem T(n) a T(n/b) + O(n d ) We will assume for convenience that n is a power of b. So, the following will ignore rounding in n/b Since subproblem size decreases by factor of b with each level of recursion, we reach the base case (size = 1) after log b n levels Since the branching factor = a, at the k th level of the tree of recursive calls, there are a k subproblems, each of size n/b k

20 Anatomy of the Master Theorem T(n) a T(n/b) + O(n d ) Accounting When solving an instance of size = n, we pay O(n d ) work. Work done inside the a recursive calls of instance size n/b is accounted for with those calls. From previous slide: at the k th level of this tree of recursive calls, there are a k subproblems, each of size n/b k Total work done at the k th level: a k x O(n/b k ) d = O(n d ) x (a / b d ) k

21 Anatomy of the Master Theorem T(n) a T(n/b) + O(n d ) Total work done at the k th level: a k x O(n/b k ) d = O(n d ) x (a / b d ) k To add up total work done, we sum up over all levels This means summing a geometric series with ratio a / b d Terms of the series are of form (a / b d ) k We sum over all levels from k = 0 (at root of tree) to k = log b n (at the leaves)

22 Anatomy of the Master Theorem Summing A Geometric Series We want to evaluate: k = 0,, logb n O(n d ) x (a / b d ) k = O(n d ) x k = 0,, logb n (a / b d ) k If (a / b d ) < 1, the series converges and k = 0,, logb n (a / b d ) k is bounded by a constant. Letting r = a / b d, k = 0,, logb n r k k = 0,, r k = 1/(1 r) So, k = 0,, logb n O(n d ) x (a / b d ) k = O(n d )

23 Anatomy of the Master Theorem Summing A Geometric Series We want to evaluate: k = 0,, logb n O(n d ) x (a / b d ) k If (a / b d ) = 1, the sum is O(n d ) x log b n

24 Anatomy of the Master Theorem Summing A Geometric Series We want to evaluate: k = 0,, logb n O(n d ) x (a / b d ) k If (a / b d ) > 1, the series diverges and the sum is dominated by the last term. We get O(n d ) x (a / b d ) log bn = O(n d ) x (a log bn ) / (b d ) log bn = O(n d ) x (a log bn ) / (b d ) log bn See A Fact About Logarithms, next slide = O(n d ) x (a log bn ) / b d log bn = O(n d ) x (a log bn ) / n d = O(n log ba )

25 Proof of Master Theorem for Recurrences Recurrence T(n) a T(n/b) + O(n d ) 1) T(n) = O(n d ) if a < b d 2) T(n) = O(n d log n) if a = b d 3) T(n) = O(n log ba ) if a > b d Assume n is a power of b can ignore rounding in n/b Subproblem size decreases by factor of b with each level of recursion reaches base case after log b n levels Branching factor a k th level of tree has a k subproblems each of size n/b k Total work done at k th level is a k x O(n/b k ) d = O(n d ) x (a / b d ) k

26 Proof of Master Theorem for Recurrences Recurrence T(n) a T(n/b) + O(n d ) 1) T(n) = O(n d ) if a < b d 2) T(n) = O(n d log n) if a = b d 3) T(n) = O(n log ba ) if a > b d Total work done at k th level is a k x O(n/b k ) d = O(n d ) x (a / b d ) k As k goes from 0 (root) to log b n (leaves), have geometric series with ratio a / b d a / b d < 1 sum is O(n d ) a / b d > 1 sum is given by last term, O(n log ba ) a / b d = 1 sum is given by O(log n) terms equal to O(n d )

27 Discussion of Lower Bounds Lower bounds are useful Know that you can t do any better than a certain algorithmic (time, space, etc.) complexity Lower bounds can be migrated from one problem to another, via a type of argument called reduction Roughly: We know that problem A has complexity lower bound X. But, I can solve any instance of A using a solver for problem B. Therefore, problem B must also have complexity lower bound X. We will see a lot of reductions if/when we discuss NP-completeness later this quarter

28 Convex Hull (Lecture 1, Slide 32) Given a planar pointset S, its convex hull consists of all extreme points, i.e., points of S that cannot be expressed as convex combinations of any other points of S If you put a rubber band around the pointset, where does it snap tight?

29 Convex Hull Task: Return points in convex hull of S in order What is a lower bound on complexity? Q: If sorting requires (n log n) time, then why MUST convex hull also require (n log n) time? // will return to this How would you go about finding the convex hull?

30 Convex Hull (1) Task: Return points in convex hull of S in order Many known approaches: Giftwrap O(n 2 ) (O(nh)) Graham Scan O(n log n) D/Q O(n log n) Algorithm by Preparata and Hong left-right split find convex hulls merge convex hulls

31 Convex Hull (2) Task: Return points in convex hull of S in order Many known approaches: Giftwrap O(n 2 ) (O(nh)) Graham Scan O(n log n) D/Q O(n log n) Algorithm by Preparata and Hong left-right split find convex hulls merge convex hulls

32 Convex Hull (3) Task: Return points in convex hull of S in order Many known approaches: Giftwrap O(n 2 ) (O(nh)) Graham Scan O(n log n) D/Q O(n log n) Algorithm by Preparata and Hong left-right split find convex hulls merge convex hulls

33 A Lower Bound on Convex Hull Task: sort the set of numbers {3,1,4,5,2} Task: find the convex hull of the set of points {(3,9), (1,1), (4,16), (5,25), (2,4)}

34 A Lower Bound on Convex Hull Given: Sorting LB is (n log n) Reduction from Sorting to Convex Hull Input: an arbitrary instance I SORT of SORT Transform I SORT = {x 1, x 2,, x n } into an instance I C-HULL = {(x 1,x 12 ), (x 2,x 22 ),, (x n,x n2 )} of C-HULL Solve the C-HULL problem for instance I C-HULL Transform solution of I C-HULL to solution of I SORT

35 A Lower Bound on Convex Hull Given: Sorting LB is (n log n) Reduction from Sorting to Convex Hull Input: an arbitrary instance I SORT of SORT Transform I SORT = {x 1, x 2,, x n } into an instance I C-HULL = {(x 1,x 12 ), (x 2,x 22 ),, (x n,x n2 )} of C-HULL Solve the C-HULL problem for instance I C-HULL Transform solution of I C-HULL to solution of I SORT SORT input TX O(n) C-HULL input C-HULL solver C-HULL solution TX O(n) SORT solution SORT solver (n log n) Note: Each Transform has O(n) complexity C-HULL solver cannot be faster than (n log n)

36 The Closest Pair Problem Closest Pair: Given n points in the plane, return the closest pair Naïve (n 2 ) D/Q Approach: (n log n) (1) Split into two pointsets S 1, S 2 by x-coord natural order (2) Find closest pair distances d 1, d 2 in S 1, S 2 respectively without loss of generality, can assume d 1 <d 2 (3) Merge the solutions Do we have to check all s 1 -s 2 pairs, s 1 S 1, s 2 S 2? What if there are lots of points in middle strip? Key: Step (3) Observation: There are at most O(1) points in middle strip with y d 1

37 DQ Closest Pair (1) sort by x- and y-coords tool: one-time sort O(n log n) (2) solve subproblems of size n/2 2T(n/2) (3) eliminate points outside strips O(n) (4) sort by y-coord within strips O(n) (5) compare each point to 6 neighbors O(n) Complexity O(n log n) + T(n) = 2T(n/2) + O(n) T(2) = 1 T(n) O(n log n) Again, with one-time sort, can always output subproblems in y-sorted order

38 DQ Closest Pair Comment Note that we exploited geometry Example: what is the maximum possible degree of a vertex in a minimum spanning tree? If the problem is geometric? planar pointset If the problem is non-geometric? edge-weighted graph

39 The Selection Problem The SELECTION problem Given a list L and a number k Select (L, k) returns the k th smallest element of L What is an efficient algorithm? Note: if k = L /2, then Select(L,k) returns the median Note: Sorting + finding k th smallest O(n log n) Can do better using DQ (maybe, recalling idea of QSort pivot )

40 DQ Selection Pick a value v Split S into three parts: S <v, S =v, S >v Select (S,k) = Select(S <v,k) if k S <v (look in left) v if S <v < k S <v + S =v Select(S >v,k - S <v - S =v ) if k > S <v + S =v (look in right) Single split operation effectively reduces size of search space balanced By how much? Depends on v!!! If we could magically pick v so that S <v, S >v S /2 then we would have T(n) T(n/2) + O(n) T(n) = O(n) Time on array of size n Time for Split

41 EXTRA SLIDES

42 Sorting (With Comparisons) Input: sequence of numbers Output: a sorted sequence Observe: Sorting == Identifying a Permutation KEY OBSERVATIONS B < C Y A < B N A < C 1. We need at least as many leaves in this tree as there are possible outcomes (= n! permutations of n elements) 2. The number of comparisons needed to get to the leaf at greatest depth from the root is the worst-case complexity of the algorithm that is embodied by this comparison tree Y N Y N A < B < C A < C B < A < C B < C Y N Y N A < C < B C < A < B B < C < A C < B < A

43 Searching an Ordered List Input: ordered list L of n numbers, and a target number x Output: Find x if it exists in L Observe: Searching == Identifying the index of x in L Y Y x < L[n/4] N x < L[n/2] N x < L[3n/4] Y N KEY OBSERVATIONS 1. We generally use binary search, which takes log(n) time in the worst case. Why is binary search the best possible strategy? (*) 2. The number of comparisons needed to get to the leaf at greatest depth from the root is the worst-case complexity of the algorithm that is embodied by this comparison tree 3. Need at least n leaves 4. Height of comparison tree is (log n) (*) Interpolation Search is better than binary search when we know something about the distribution of the elements we are searching. For example, when humans use Interpolation Search when looking up something in a dictionary or a phone book.

44 A Lower Bound on Sorting Complexity In the comparison model of computation, can we find a lower bound on the complexity of sorting? From last slide: Sorting Identifying Permutation Binary Tree: Root at level (height) 0 Theorem: There exists some c > 0 such that for all algorithms which use comparisons to sort, and for all input sizes n, at least one input requires cn log n comparisons Fact: Binary tree of height h has at most 2 h leaves Observation from last slide: n! leaves needed comparison tree must have h log(n!) h is maximum (= worst-case) #comparisons needed to sort input of size n using the corresponding algorithm

45 Sorting Lower Bound (DETAIL) Goal: log(n!) (n log n) Claim: log(n!) O(n log n) n! n n log n! n log n Claim: log(n!) (n log n) n! (n/2) n/2 log n! n/2 log(n/2) 2 log n! n log (n/2) Observe: log (n/2) + 1 = log n 2 log (n/2) log n n > 2 4 log n! n 2 log (n/2) 4 log n! n log n

46 What About LB For the Average Case? Of possible interest: Worst-case analysis can be uninformative QuickSort: worst case is n 2 Simplex Method for linear programming: worst case is exponential Can we lower-bound average case complexity? First question: Is average case well-defined? Want i p i d i expected depth of a leaf in the comparison tree d i depth of leaf i; i = input with probability = p i Assume all input permutations equally probable ( equiprobable )

47 Average-Case Complexity of Sorting Q: Even though all sorting algs have some input which requires n log n time, is there an algorithm with better than (n log n) average-case performance? Theorem: If all n! input permutations equiprobable, then any decision tree that sorts has expected depth (n log n). Let D(m) smallest sum of leaf depths over all binary trees with m leaves Claim: D(m) m log m. If Claim true, use m = n! and fact that log n! (n log n) D(n!) n! log n! average leaf depth is (n log n)

48 Average-Case Complexity of Sorting (cont.) Claim: D(m) m log m Proof by induction on m. (D(T) sum of leaf depths of tree T, where unambiguous.) Claim trivial for m = 1; assume Claim m < k (strong I.H.). Any tree T with k leaves can be viewed as a root and two subtrees T i and T k-i (with i and k-i leaves respectively) D(T) = i + D(T i ) + (k-i) + D(T k-i ) D(k) = min 1 i k [k + D(i) + D(k-i)] k + min i [D(i) + D(k-i)] k + min i [i log i + (k-i) log (k-i)] (by I.H.) which is minimized for i = k/2. D(k) k + k log (k/2) = k + k(log k 1) = k log k