CSE 421 Algorithms: Divide and Conquer

Transcription

1 CSE 42 Algorithms: Divide and Conquer Larry Ruzzo Thanks to Richard Anderson, Paul Beame, Kevin Wayne for some slides

2 Outline: General Idea algorithm design paradigms: divide and conquer Review of Merge Sort Why does it work? Importance of balance Importance of super-linear growth Some interesting applications Inversions Closest points Integer Multiplication Finding & Solving Recurrences 2

3 Divide & Conquer algorithm design techniques Reduce problem to one or more sub-problems of the same type Typically, each sub-problem is at most a constant fraction of the size of the original problem Subproblems typically disjoint Often gives significant, usually polynomial, speedup Examples: Binary Search, Mergesort, Quicksort (roughly), Strassen s Algorithm, integer multiplication, powering, FFT, 3

4 Motivating Example: Mergesort 4

5 merge sort MS(A: array[..n]) returns array[..n] { If(n=) return A; New U:array[:n/2] = MS(A[..n/2]); New L:array[:n/2] = MS(A[n/2+..n]); Return(Merge(U,L)); } A U C Merge(U,L: array[..n]) { New C: array[..2n]; a=; b=; For i = to 2n C[i] = smaller of U[a], L[b] and correspondingly a++ or b++, while being careful about running past end of either ; Return C; } Time: Θ(n log n) L split sort merge 5

6 Why does it work? Suppose we ve already invented DumbSort, taking time n 2 Try Just One Level of divide & conquer: DumbSort(first n/2 elements) O((n/2) 2 ) DumbSort(last n/2 elements) O((n/2) 2 ) Merge results divide & conquer the key idea O(n) Time: 2 (n/2) 2 + n = n 2 /2 + n n 2 Almost twice as fast! D&C in a nutshell 6

7 d&c approach, cont. Moral : two halves are better than a whole Two problems of half size are better than one full-size problem, even given O(n) overhead of recombining, since the base algorithm has super-linear complexity. Moral 2: If a little's good, then more's better Two levels of D&C would be almost 4 times faster, 3 levels almost 8, etc., even though overhead is growing. Best is usually full recursion down to some small constant size (balancing "work" vs "overhead"). In the limit: you ve just rediscovered mergesort! 7

8 d&c approach, cont. Moral 3: unbalanced division good, but less so: (.n) 2 + (.9n) 2 + n =.82n 2 + n The 8% savings compounds significantly if you carry recursion to more levels, actually giving O(nlogn), but with a bigger constant. So worth doing if you can t get 5-5 split, but balanced is better if you can. This is intuitively why Quicksort with random splitter is good badly unbalanced splits are rare, and not instantly fatal. Moral 4: but consistent, completely unbalanced division doesn t help much: () 2 + (n-) 2 + n = n 2 - n + 2 Little improvement here. 8

9 mergesort (review) Mergesort: (recursively) sort 2 half-lists, then merge results. T(n) = 2T(n/2)+cn, n 2 T() = Solution: Θ(n log n) (details later) Log n levels O(n) work per level 9

10 Example: Counting Inversions

11 Let a,... a n be a permutation of.. n (a i, a j ) is an inversion if i < j and a i > a j Inversion Problem 4, 6,, 7, 3, 2, 5 Problem: given a permutation, count the number of inversions This can be done easily in O(n 2 ) time Can we do better?

12 Application Counting inversions can be use to measure closeness of ranked preferences People rank 2 movies, based on their rankings you cluster people who like the same types of movies Can also be used to measure nonlinear correlation 2

13 Let a,... a n be a permutation of.. n (a i, a j ) is an inversion if i < j and a i > a j Inversion Problem 4, 6,, 7, 3, 2, 5 Problem: given a permutation, count the number of inversions This can be done easily in O(n 2 ) time Can we do better? 3

14 Counting Inversions Count inversions on left half Count inversions on right half Count the inversions between the halves 4

15 Count the Inversions

16 Can we count inversions between sub-problems in O(n) time? Yes Count inversions while merging Standard merge algorithm add to inversion count when an element is moved from the right array to the solution. (Add how much? Why not left array?) 6

17 Counting inversions while merging Indicate the number of inversions for each element detected when merging 7

18 Inversions Counting inversions between two sorted lists O() per element to count inversions x x x x x x x x y y y y y y y y z z z z z z z z z z z z z z z z Algorithm summary Satisfies the Standard recurrence T(n) = 2 T(n/2) + cn 8

19 A Divide & Conquer Example: Closest Pair of Points 9

20 closest pair of points: non-geometric version Given n points and arbitrary distances between them, find the closest pair. (E.g., think of distance as airfare definitely not Euclidean distance!) ( and all the rest of the ( n ) edges ) 2 Must look at all n choose 2 pairwise distances, else any one you didn t check might be the shortest. Also true for Euclidean distance in -2 dimensions? 2

21 closest pair of points: dimensional version Given n points on the real line, find the closest pair Closest pair is adjacent in ordered list Time O(n log n) to sort, if needed Plus O(n) to scan adjacent pairs Key point: do not need to calc distances between all pairs: exploit geometry + ordering 2

22 closest pair of points: 2 dimensional version Closest pair. Given n points in the plane, find a pair with smallest Euclidean distance between them. Fundamental geometric primitive. Graphics, computer vision, geographic information systems, molecular modeling, air traffic control. Special case of nearest neighbor, Euclidean MST, Voronoi. fast closest pair inspired fast algorithms for these problems Brute force. Check all pairs of points p and q with Θ(n 2 ) comparisons. -D version. O(n log n) easy if points are on a line. Assumption. No two points have same x coordinate. Just to simplify presentation 22

23 closest pair of points. 2d, Euclidean distance: st try Divide. Sub-divide region into 4 quadrants. 23

24 closest pair of points: st try Divide. Sub-divide region into 4 quadrants. Obstacle. Impossible to ensure n/4 points in each piece, so the balanced subdivision goal may be elusive/problematic. 24

25 Algorithm. closest pair of points Divide: draw vertical line L with n/2 points on each side. L 25

26 Algorithm. closest pair of points Divide: draw vertical line L with n/2 points on each side. Conquer: find closest pair on each side, recursively. L

27 Algorithm. closest pair of points Divide: draw vertical line L with n/2 points on each side. Conquer: find closest pair on each side, recursively. Combine: find closest pair with one point in each side. Return best of 3 solutions. L seems like Θ(n 2 )?

28 closest pair of points Find closest pair with one point in each side, assuming distance < δ. L 2 2 δ = min(2, 2) 28

29 closest pair of points Find closest pair with one point in each side, assuming distance < δ. Observation: suffices to consider points within δ of line L. L 2 2 δ = min(2, 2) 29 δ

30 closest pair of points Find closest pair with one point in each side, assuming distance < δ. Observation: suffices to consider points within δ of line L. Almost the one-d problem again: Sort points in 2δ-strip by their y coordinate. 7 L δ = min(2, 2) 2 δ 3

31 closest pair of points Find closest pair with one point in each side, assuming distance < δ. Observation: suffices to consider points within δ of line L. Almost the one-d problem again: Sort points in 2δ-strip by their y coordinate. Only check pts within 8 in sorted list! 7 L δ = min(2, 2) 2 δ 3

32 Def. Let s i have the i th smallest y-coordinate among points in the 2δ-width-strip. Claim. If j i 8, then the distance between s i and s j is > δ. Pf: No two points lie in the same δ/2-by-δ/2 square: closest pair of points i j δ/2 δ/2 " δ % $ ' # 2& 2 " + δ % $ ' # 2& 2 = 2 2 δ.7δ < δ so 7 points within +δ of y(s i ). 32 δ δ

33 closest pair algorithm Closest-Pair(p,, p n ) { if(n <=??) return?? Compute separation line L such that half the points are on one side and half on the other side. δ = Closest-Pair(left half) δ 2 = Closest-Pair(right half) δ = min(δ, δ 2 ) Delete all points further than δ from separation line L Sort remaining points p[] p[m] by y-coordinate. for i =..m k = while i+k <= m && p[i+k].y < p[i].y + δ δ = min(δ, distance between p[i] and p[i+k]); k++; } return δ. 33

34 closest pair of points: analysis Analysis, I: Let D(n) be the number of pairwise distance calculations in the Closest-Pair Algorithm when run on n points D(n) # n =& $ % 2D( n /2) + 7n n > ' ( D(n) = O(n log n) BUT that s only the number of distance calculations What if we counted comparisons? 34

35 closest pair of points: analysis Analysis, II: Let C(n) be the number of comparisons between coordinates/distances in the Closest-Pair Algorithm when run on n points " $ C(n) # %$ 2C n / 2 for some constant k n = ( ) + kn logn n > Q. Can we achieve O(n log n)? & $ ' ($ C(n) = O(n log 2 n) A. Yes. Don't sort points from scratch each time. Sort by x at top level only. Each recursive call returns δ and list of all points sorted by y Sort by merging two pre-sorted lists. T(n) 2T( n/2) + O(n) T(n) = O(n log n) 35

36 Code is longer & more complex is it worth the effort? O(n log n) vs O(n 2 ) may hide x in constant? How many points? n Speedup: n 2 / ( n log 2 n).3.5,, 75, 62,, 5,7,, 43,4 36

37 Going From Code to Recurrence 37

38 going from code to recurrence Carefully define what you re counting, and write it down! Let C(n) be the number of comparisons between sort keys used by MergeSort when sorting a list of length n In code, clearly separate base case from recursive case, highlight recursive calls, and operations being counted. Write Recurrence(s) 38

39 merge sort Base Case MS(A: array[..n]) returns array[..n] { If(n=) return A; New L:array[:n/2] = MS(A[..n/2]); New R:array[:n/2] = MS(A[n/2+..n]); Return(Merge(L,R)); } Merge(A,B: array[..n]) { New C: array[..2n]; a=; b=; For i = to 2n { C[i] = smaller of A[a], B[b] and a++ or b++ ; Return C; } Recursive calls One Recursive Level Operations being counted 39

40 the recurrence Base case C(n) = # if n = $ % 2C(n /2) + (n ) if n > Recursive calls Total time: proportional to C(n) (loops, copying data, parameter passing, etc.) One compare per element added to merged list, except the last. 4

41 going from code to recurrence Carefully define what you re counting, and write it down! Let D(n) be the number of pairwise distance calculations in the Closest-Pair Algorithm when run on n points In code, clearly separate base case from recursive case, highlight recursive calls, and operations being counted. Write Recurrence(s) 4

42 Basic operations: distance calcs closest pair algorithm Closest-Pair(p,, p n ) { if(n <= ) return Base Case Compute separation line L such that half the points are on one side and half on the other side. δ = Closest-Pair(left half) δ 2 = Closest-Pair(right half) δ = min(δ, δ 2 ) Recursive calls (2) 2D(n / 2) } Delete all points further than δ from separation line L Sort remaining points p[] p[m] by y-coordinate. for i =..m k = while i+k <= m && p[i+k].y < p[i].y + δ return δ. Basic operations at this recursive level δ = min(δ, distance between p[i] and p[i+k]); k++; One recursive level 7n 42

43 closest pair of points: analysis Analysis, I: Let D(n) be the number of pairwise distance calculations in the Closest-Pair Algorithm when run on n points D(n) # n =& $ % 2D( n /2) + 7n n > ' ( D(n) = O(n log n) BUT that s only the number of distance calculations What if we counted comparisons? 43

44 going from code to recurrence Carefully define what you re counting, and write it down! Let D(n) be the number of comparisons between coordinates/distances in the Closest-Pair Algorithm when run on n points In code, clearly separate base case from recursive case, highlight recursive calls, and operations being counted. Write Recurrence(s) 44

45 Basic operations: comparisons Base Case closest pair algorithm Closest-Pair(p,, p n ) { if(n <= ) return Recursive calls (2) } Compute separation line L such that half the points are on one side and half on the other side. δ = Closest-Pair(left half) δ 2 = Closest-Pair(right half) δ = min(δ, δ 2 ) Delete all points further than δ from separation line L Sort remaining points p[] p[m] by y-coordinate. for i =..m Basic operations at this recursive level k = while i+k <= m && p[i+k].y < p[i].y + δ δ = min(δ, distance between p[i] and p[i+k]); k++; return δ. k n log n 2C(n / 2) k 2 n k 3 n log n 8n 7n One recursive level 45

46 closest pair of points: analysis Analysis, II: Let C(n) be the number of comparisons of coordinates/distances in the Closest-Pair Algorithm when run on n points C(n) 2C n / 2 for k 4 = k + k 2 + k 3 +6 n = ( ) + k 4 n log 2 n n > Q. Can we achieve time O(n log n)? C(n) = O(n log 2 n) A. Yes. Don't sort points from scratch each time. Sort by x at top level only. Each recursive call returns δ and list of all points sorted by y Sort by merging two pre-sorted lists. T(n) 2T( n/2) + O(n) T(n) = O(n log n) 46

47 Integer Multiplication 47

48 integer arithmetic Add. Given two n-bit integers a and b, compute a + b. O(n) bit operations. Multiply. Given two n-digit integers a and b, compute a b. The grade school method: Θ(n 2 ) bit operations Add * Multiply

49 integer arithmetic Add. Given two n-bit integers a and b, compute a + b. O(n) bit operations. Multiply. Given two n-bit integers a and b, compute a b. The grade school method: Θ(n 2 ) bit operations Add * Multiply Carries

50 divide & conquer multiplication: warmup To multiply two 2-digit integers: Multiply four -digit integers. Add, shift some 2-digit integers to obtain result. x = x + x y = y + y xy = x + x ( ) ( y + y ) ( ) + x y = x y + x y + x y y y x x x y x y Same idea works for long integers can split them into 4 half-sized ints ( becomes k, k = length/2) x y x y 5

51 divide & conquer multiplication: warmup To multiply two n-bit integers: Multiply four ½n-bit integers. Shift/add four n-bit integers to obtain result. x = 2 n / 2 x + x y = 2 n / 2 y + y xy = 2 n / 2 x + x ( ) ( 2 n / 2 y + y ) * y y x x x y ( ) + x y = 2 n x y + 2 n / 2 x y + x y x y x y ( ) T(n) = 4T n /2 + Θ(n) T(n) = Θ(n2 ) recursive calls add, shift x y assumes n is a power of 2 5

52 key trick: 2 multiplies for the price of : x = 2 n / 2 x + x y = 2 n / 2 y + y xy = 2 n / 2 x + x ( ) ( 2 n / 2 y + y ) Well, ok, 4 for 3 is more accurate ( ) + x y = 2 n x y + 2 n / 2 x y + x y α = x + x β = y + y αβ = ( x + x ) ( y + y ) = x y + x y + x y ( x y + x y ) = αβ x y x y ( ) + x y 52

53 To multiply two n-bit integers: Add two pairs of ½n bit integers. Multiply three pairs of ½n-bit integers. Add, subtract, and shift n-bit integers to obtain result. Karatsuba multiplication x = 2 n /2 x + x y = 2 n /2 y + y xy = 2 n x y + 2 n /2 x y + x y ( ) + x y ( ) + x y = 2 n x y + 2 n /2 (x + x )(y + y ) x y x y A B A C C Theorem. [Karatsuba-Ofman, 962] Can multiply two n-digit integers in O(n.585 ) bit operations. ( # $ ) + T % n /2& ( ) + T( + % n /2& ) T(n) T n /2 recursive calls Sloppy version : T(n) 3T(n /2) + O(n) T(n) = O(n log 2 3 ) = O(n.585 ) + Θ(n) add, subtract, shift 53

54 Karatsuba multiplication Theorem. [Karatsuba-Ofman, 962] Can multiply two n-digit integers in O(n.585 ) bit operations. ( # $ ) + T % n /2& ( ) + T( + % n /2& ) T(n) T n /2 recursive calls Sloppy version : T(n) 3T(n /2) + O(n) T(n) = O(n log 2 3 ) = O(n.585 ) + Θ(n) add, subtract, shift Best: Solve the exact recurrence 2 nd best: Solve the sloppy version Almost always gives the right asymptotics Alternatively, you can often change the algorithm to simplify the recurrence so that you can solve it. E.g., sloppy = exact if Get rid of and by padding n to 2 log n 2 Get rid of + n/2 by peeling off one bit: T(n/2)+O(n) 54

55 Karatsuba multiplication Theorem. [Karatsuba-Ofman, 962] Can multiply two n-digit integers in O(n.585 ) bit operations. ( # $ ) + T % n /2& ( ) + T( + % n /2& ) T(n) T n /2 recursive calls Sloppy version : T(n) 3T(n /2) + O(n) T(n) = O(n log 2 3 ) = O(n.585 ) + Θ(n) add, subtract, shift Alternatively, you can often change the algorithm to simplify the recurrence so that you can solve it. E.g., sloppy becomes exact if Kill & : pad n to 2 log 2 n Kill + n/2 : peel off one bit: T(n/2)+O(n) x y x + xy zzz 55

56 multiplication the bottom line Naïve: Θ(n 2 ) Karatsuba: Θ(n.59 ) Amusing exercise: generalize Karatsuba to do 5 size n/3 subproblems Θ(n.46 ) Best known: Θ(n log n loglog n) "Fast Fourier Transform" but mostly unused in practice (unless you need really big numbers - a billion digits of π, say) High precision arithmetic IS important for crypto, among other uses 56

57 Recurrences Above: Where they come from, how to find them Next: how to solve them 57

58 mergesort (review) Mergesort: (recursively) sort 2 half-lists, then merge results. T(n) = 2T(n/2)+cn, n 2 T() = Solution: Θ(n log n) (details later) now! Log n levels O(n) work per level 58

59 Solve: T() = c T(n) = 2 T(n/2) + cn n = 2 k ; k = log 2 n Level Num Size Work = 2 =2 n n cn cn 2 = 2 n/2 2cn/2 2 4 = 2=2 2 2 n/2 n/4 2 c 4cn/4 n/2 2 4=2 2 n/4 4 c n/4 i 2 i n/2 i 2 i c n/2 i i 2 i n/2 i 2 i c n/2 i k- 2 k- n/2 k- 2 k- c n/2 k- Level Num Size Work k- 2 k- n/2 k- 2 k- c n/2 k- k 2 k n/2 k = 2 k T() Total Work: c n (+log 2 n) (add last col) 59

60 Solve: T() = c T(n) = 4 T(n/2) + cn n = 2 k ; k = log 2 n Level Num Size Work = =4 4 n n cn cn 4 = 4=4 4 n/2 4 c 4cn/ =4 = n/4 6 6cn/4 c i i 4 i 4 i n/2 i 4 i c 4 i n/2 c n/2 i k- 4 k- n/2 k- 4 k- c n/2 k- k k 4 k 4 k n/2 n/2 = k = 4 k T() 4 k T() Level Num Size Work k- 4 k- n/2 k- 4 k- c n/2 k- Total Work: T(n) = k 4 i cn / 2 i = O(n 2 ) i = 4 k = (2 2 ) k = 6 (2 k ) 2 = n 2

61 Solve: T() = c T(n) = 3 T(n/2) + cn n = 2 k ; k = log 2 n Total Work: T(n) = k i = Level Num Size Work =3 n cn 3=3 n/2 3 c n/2 2 9=3 2 n/4 9 c n/4 i 3 i n/2 i 3 i c n/2 i k- 3 k- n/2 k- 3 k- c n/2 k- k 3 k n/2 k = 3 k T() Level Num Size Work = 3 n cn 3 = 3 n/2 3cn/2 2 9 = 3 2 n/4 9cn/4 i 3 i n/2 i 3 i c n/2 i k- 3 k- n/2 k- 3 k- c n/2 k- k 3 k n/2 k = 3 k T() 3 i cn / 2 i 6

62 a useful identity Theorem: for x, + x + x 2 + x x k = (x k+ -)/(x-) proof: y = + x + x 2 + x x k xy = x + x 2 + x x k + x k+ xy-y = x k+ - y(x-)= x k+ - y = (x k+ -)/(x-) 62

63 Solve: T() = c T(n) = 3 T(n/2) + cn (cont.) T(n) = k 3 i cn / 2 i i= = cn k 3 i / 2 i i= k i= = cn 3 2 ( ) i ( 2 = cn 3 ) k+ ( )

64 Solve: T() = c T(n) = 3 T(n/2) + cn (cont.) ( 3 ) k+ cn 2 ( ) = 2cn ( 2 3 ) k+ 3 2 ( ) < 2cn( 2 3 ) k+ = 3cn( 2 3 ) k = 3cn 3k 2 k 64

65 Solve: T() = c T(n) = 3 T(n/2) + cn (cont.) 3cn 3k 2 k = 3cn 3log2 n 2 log 2 n = 3cn 3log 2 n n = 3c3 log 2 n = 3c( n log 2 3 ) = O( n ) a log b n = ( b log b a ) log b n = ( b log b n ) log b a = n log b a 65

66 T(n) = at(n/b)+cn k for n > b then divide and conquer master recurrence a > b k T(n) = Θ(n log b a ) [many subprobs leaves dominate] a < b k T(n) = Θ(n k ) [few subprobs top level dominates] a = b k T(n) = Θ (n k log n) [balanced all log n levels contribute] Fine print: T() = d; a ; b > ; c, d, k ; n = b t for some t > ; a, b, k, t integers. True even if it is n/b instead of n/b. 66

67 master recurrence: proof sketch Expand recurrence as in earlier examples, to get T(n) = n h ( d + c S ) where h = log b (a) (and n h = number of tree leaves) and, where x = b k /a. If c = the sum S is irrelevant, and T(n) = O(n h ): all work happens in the base cases, of which there are n h, one for each leaf in the recursion tree. If c >, then the sum matters, and splits into 3 cases (like previous slide): if x <, then S < x/(-x) = O(). if x =, then S = log b (n) = O(log n). S = [S is the first log n terms of the infinite series with that sum.] [All terms in the sum are and there are that many terms.] if x >, then S = x (x +log b (n) -)/(x-). [And after some algebra, n h * S = O(n k ).] log b n x j j= 67

68 Another Example: Exponentiation 68

69 another d&c example: fast exponentiation Power(a,n) Input: integer n and number a Output: a n Obvious algorithm n- multiplications Observation: if n is even, n = 2m, then a n = a m a m 69

70 Power(a,n) if n = then return() if n = then return(a) x Power(a, n/2 ) x x x if n is odd then x a x return(x) divide & conquer algorithm 7

71 Let M(n) be number of multiplies Worst-case recurrence: By master theorem M(n) = O(log n) & ( M(n) = ' )( More precise analysis: (a=, b=2, k=) analysis n M ("# n / 2$% ) + 2 n > M(n) = log 2 n + (# of s in n s binary representation) - Time is O(M(n)) if numbers < word size, else also depends on length, multiply algorithm 7

72 a practical application - RSA Instead of a n want a n mod N a i+j mod N = ((a i mod N) (a j mod N)) mod N same algorithm applies with each x y replaced by ((x mod N) (y mod N)) mod N In RSA cryptosystem (widely used for security) need a n mod N where a, n, N each typically have 24 bits Power: at most 248 multiplies of 24 bit numbers relatively easy for modern machines Naive algorithm: 2 24 multiplies 72

73 Utility: Idea: Correctness often easy; often faster Two halves are better than a whole if the base algorithm has super-linear complexity. If a little's good, then more's better repeat above, recursively d & c summary Analysis: recursion tree or Master Recurrence among others Applications: Many. Binary Search, Merge Sort, (Quicksort), Closest Points, Integer Multiply, Exponentiation, 73