Kartsuba s Algorithm and Linear Time Selection

Transcription

1 CS 374: Algorithms & Models of Computation, Fall 2015 Kartsuba s Algorithm and Linear Time Selection Lecture 09 September 22, 2015 Chandra & Manoj (UIUC) CS374 1 Fall / 32

2 Part I Fast Multiplication Chandra & Manoj (UIUC) CS374 2 Fall / 32

3 Multiplying Numbers Problem Given two n-digit numbers x and y, compute their product. Grade School Multiplication Compute partial product by multiplying each digit of y with x and adding the partial products Chandra & Manoj (UIUC) CS374 3 Fall / 32

4 Time Analysis of Grade School Multiplication 1 Each partial product: Θ(n) 2 Number of partial products: Θ(n) 3 Addition of partial products: Θ(n 2 ) 4 Total time: Θ(n 2 ) Chandra & Manoj (UIUC) CS374 4 Fall / 32

5 A Trick of Gauss Carl Friedrich Gauss: Prince of Mathematicians Observation: Multiply two complex numbers: (a + bi) and (c + di) (a + bi)(c + di) = ac bd + (ad + bc)i Chandra & Manoj (UIUC) CS374 5 Fall / 32

6 A Trick of Gauss Carl Friedrich Gauss: Prince of Mathematicians Observation: Multiply two complex numbers: (a + bi) and (c + di) (a + bi)(c + di) = ac bd + (ad + bc)i How many multiplications do we need? Chandra & Manoj (UIUC) CS374 5 Fall / 32

7 A Trick of Gauss Carl Friedrich Gauss: Prince of Mathematicians Observation: Multiply two complex numbers: (a + bi) and (c + di) (a + bi)(c + di) = ac bd + (ad + bc)i How many multiplications do we need? Only 3! If we do extra additions and subtractions. Compute ac, bd, (a + b)(c + d). Then (ad + bc) = (a + b)(c + d) ac bd Chandra & Manoj (UIUC) CS374 5 Fall / 32

8 Divide and Conquer Assume n is a power of 2 for simplicity and numbers are in decimal. Split each number into two numbers with equal number of digits 1 x = x n 1 x n 2... x 0 and y = y n 1 y n 2... y 0 2 x = x n 1... x n/ x n/ x 0 3 x L = 10 n/2 x L where x L = x n 1... x n/2 and x R = x n/ x 0 4 Similarly y = 10 n/2 y L + y R where y L = y n 1... y n/2 and y R = y n/ y 0 Chandra & Manoj (UIUC) CS374 6 Fall / 32

9 Example = ( ) ( ) = ( ) Chandra & Manoj (UIUC) CS374 7 Fall / 32

10 Divide and Conquer Assume n is a power of 2 for simplicity and numbers are in decimal. 1 x = x n 1 x n 2... x 0 and y = y n 1 y n 2... y 0 2 x = 10 n/2 x L + x R where x L = x n 1... x n/2 and x R = x n/ x 0 3 y = 10 n/2 y L + y R where y L = y n 1... y n/2 and Therefore y R = y n/ y 0 xy = (10 n/2 x L + x R )(10 n/2 y L + y R ) = 10 n x L y L + 10 n/2 (x L y R + x R y L ) + x R y R Chandra & Manoj (UIUC) CS374 8 Fall / 32

11 Time Analysis xy = (10 n/2 x L + x R )(10 n/2 y L + y R ) = 10 n x L y L + 10 n/2 (x L y R + x R y L ) + x R y R 4 recursive multiplications of number of size n/2 each plus 4 additions and left shifts (adding enough 0 s to the right) Chandra & Manoj (UIUC) CS374 9 Fall / 32

12 Time Analysis xy = (10 n/2 x L + x R )(10 n/2 y L + y R ) = 10 n x L y L + 10 n/2 (x L y R + x R y L ) + x R y R 4 recursive multiplications of number of size n/2 each plus 4 additions and left shifts (adding enough 0 s to the right) T(n) = 4T(n/2) + O(n) T(1) = O(1) Chandra & Manoj (UIUC) CS374 9 Fall / 32

13 Time Analysis xy = (10 n/2 x L + x R )(10 n/2 y L + y R ) = 10 n x L y L + 10 n/2 (x L y R + x R y L ) + x R y R 4 recursive multiplications of number of size n/2 each plus 4 additions and left shifts (adding enough 0 s to the right) T(n) = 4T(n/2) + O(n) T(1) = O(1) T(n) = Θ(n 2 ). No better than grade school multiplication! Chandra & Manoj (UIUC) CS374 9 Fall / 32

14 Time Analysis xy = (10 n/2 x L + x R )(10 n/2 y L + y R ) = 10 n x L y L + 10 n/2 (x L y R + x R y L ) + x R y R 4 recursive multiplications of number of size n/2 each plus 4 additions and left shifts (adding enough 0 s to the right) T(n) = 4T(n/2) + O(n) T(1) = O(1) T(n) = Θ(n 2 ). No better than grade school multiplication! Can we invoke Gauss s trick here? Chandra & Manoj (UIUC) CS374 9 Fall / 32

15 Improving the Running Time xy = (10 n/2 x L + x R )(10 n/2 y L + y R ) = 10 n x L y L + 10 n/2 (x L y R + x R y L ) + x R y R Gauss trick: x L y R + x R y L = (x L + x R )(y L + y R ) x L y L x R y R Chandra & Manoj (UIUC) CS Fall / 32

16 Improving the Running Time xy = (10 n/2 x L + x R )(10 n/2 y L + y R ) = 10 n x L y L + 10 n/2 (x L y R + x R y L ) + x R y R Gauss trick: x L y R + x R y L = (x L + x R )(y L + y R ) x L y L x R y R Recursively compute only x L y L, x R y R, (x L + x R )(y L + y R ). Chandra & Manoj (UIUC) CS Fall / 32

17 Improving the Running Time xy = (10 n/2 x L + x R )(10 n/2 y L + y R ) = 10 n x L y L + 10 n/2 (x L y R + x R y L ) + x R y R Gauss trick: x L y R + x R y L = (x L + x R )(y L + y R ) x L y L x R y R Recursively compute only x L y L, x R y R, (x L + x R )(y L + y R ). Time Analysis Running time is given by T(n) = 3T(n/2) + O(n) T(1) = O(1) which means Chandra & Manoj (UIUC) CS Fall / 32

18 Improving the Running Time xy = (10 n/2 x L + x R )(10 n/2 y L + y R ) = 10 n x L y L + 10 n/2 (x L y R + x R y L ) + x R y R Gauss trick: x L y R + x R y L = (x L + x R )(y L + y R ) x L y L x R y R Recursively compute only x L y L, x R y R, (x L + x R )(y L + y R ). Time Analysis Running time is given by T(n) = 3T(n/2) + O(n) T(1) = O(1) which means T(n) = O(n log 2 3 ) = O(n ) Chandra & Manoj (UIUC) CS Fall / 32

19 State of the Art Schönhage-Strassen 1971: O(n log n log log n) time using Fast-Fourier-Transform (FFT) Martin Fürer 2007: O(n log n2 O(log n) ) time Conjecture There is an O(n log n) time algorithm. Chandra & Manoj (UIUC) CS Fall / 32

20 Analyzing the Recurrences 1 Basic divide and conquer: T(n) = 4T(n/2) + O(n), T(1) = 1. Claim: T(n) = Θ(n 2 ). 2 Saving a multiplication: T(n) = 3T(n/2) + O(n), T(1) = 1. Claim: T(n) = Θ(n 1+log 1.5 ) Chandra & Manoj (UIUC) CS Fall / 32

21 Analyzing the Recurrences 1 Basic divide and conquer: T(n) = 4T(n/2) + O(n), T(1) = 1. Claim: T(n) = Θ(n 2 ). 2 Saving a multiplication: T(n) = 3T(n/2) + O(n), T(1) = 1. Claim: T(n) = Θ(n 1+log 1.5 ) Use recursion tree method: 1 In both cases, depth of recursion L = log n. 2 Work at depth i is 4 i n/2 i and 3 i n/2 i respectively: number of children at depth i times the work at each child 3 Total work is therefore n L i=0 2i and n L i=0 (3/2)i respectively. Chandra & Manoj (UIUC) CS Fall / 32

22 Recursion tree analysis Chandra & Manoj (UIUC) CS Fall / 32

23 Part II Selecting in Unsorted Lists Chandra & Manoj (UIUC) CS Fall / 32

24 Rank of element in an array A: an unsorted array of n integers Definition For 1 j n, element of rank j is the j th smallest element in A. Unsorted array Ranks Sort of array Chandra & Manoj (UIUC) CS Fall / 32

25 Problem - Selection Input Unsorted array A of n integers and integer j Goal Find the jth smallest number in A (rank j number) Median: j = (n + 1)/2 Chandra & Manoj (UIUC) CS Fall / 32

26 Problem - Selection Input Unsorted array A of n integers and integer j Goal Find the jth smallest number in A (rank j number) Median: j = (n + 1)/2 Simplifying assumption for sake of notation: elements of A are distinct Chandra & Manoj (UIUC) CS Fall / 32

27 Algorithm I 1 Sort the elements in A 2 Pick jth element in sorted order Time taken = O(n log n) Chandra & Manoj (UIUC) CS Fall / 32

28 Algorithm I 1 Sort the elements in A 2 Pick jth element in sorted order Time taken = O(n log n) Do we need to sort? Is there an O(n) time algorithm? Chandra & Manoj (UIUC) CS Fall / 32

29 Algorithm II If j is small or n j is small then 1 Find j smallest/largest elements in A in O(jn) time. (How?) 2 Time to find median is O(n 2 ). Chandra & Manoj (UIUC) CS Fall / 32

30 Divide and Conquer Approach 1 Pick a pivot element a from A 2 Partition A based on a. A less = {x A x a} and A greater = {x A x > a} 3 A less = j: return a 4 A less > j: recursively find jth smallest element in A less 5 A less < j: recursively find kth smallest element in A greater where k = j A less. Chandra & Manoj (UIUC) CS Fall / 32

31 Example Chandra & Manoj (UIUC) CS Fall / 32

32 Time Analysis 1 Partitioning step: O(n) time to scan A 2 How do we choose pivot? Recursive running time? Chandra & Manoj (UIUC) CS Fall / 32

33 Time Analysis 1 Partitioning step: O(n) time to scan A 2 How do we choose pivot? Recursive running time? Suppose we always choose pivot to be A[1]. Chandra & Manoj (UIUC) CS Fall / 32

34 Time Analysis 1 Partitioning step: O(n) time to scan A 2 How do we choose pivot? Recursive running time? Suppose we always choose pivot to be A[1]. Say A is sorted in increasing order and j = n. Exercise: show that algorithm takes Ω(n 2 ) time Chandra & Manoj (UIUC) CS Fall / 32

35 A Better Pivot Suppose pivot is the lth smallest element where n/4 l 3n/4. That is pivot is approximately in the middle of A Then n/4 A less 3n/4 and n/4 A greater 3n/4. If we apply recursion, Chandra & Manoj (UIUC) CS Fall / 32

36 A Better Pivot Suppose pivot is the lth smallest element where n/4 l 3n/4. That is pivot is approximately in the middle of A Then n/4 A less 3n/4 and n/4 A greater 3n/4. If we apply recursion, T(n) T(3n/4) + O(n) Implies T(n) = O(n)! Chandra & Manoj (UIUC) CS Fall / 32

37 A Better Pivot Suppose pivot is the lth smallest element where n/4 l 3n/4. That is pivot is approximately in the middle of A Then n/4 A less 3n/4 and n/4 A greater 3n/4. If we apply recursion, T(n) T(3n/4) + O(n) Implies T(n) = O(n)! How do we find such a pivot? Chandra & Manoj (UIUC) CS Fall / 32

38 A Better Pivot Suppose pivot is the lth smallest element where n/4 l 3n/4. That is pivot is approximately in the middle of A Then n/4 A less 3n/4 and n/4 A greater 3n/4. If we apply recursion, T(n) T(3n/4) + O(n) Implies T(n) = O(n)! How do we find such a pivot? Randomly? Chandra & Manoj (UIUC) CS Fall / 32

39 A Better Pivot Suppose pivot is the lth smallest element where n/4 l 3n/4. That is pivot is approximately in the middle of A Then n/4 A less 3n/4 and n/4 A greater 3n/4. If we apply recursion, T(n) T(3n/4) + O(n) Implies T(n) = O(n)! How do we find such a pivot? Randomly? In fact works! Analysis a little bit later. Chandra & Manoj (UIUC) CS Fall / 32

40 A Better Pivot Suppose pivot is the lth smallest element where n/4 l 3n/4. That is pivot is approximately in the middle of A Then n/4 A less 3n/4 and n/4 A greater 3n/4. If we apply recursion, T(n) T(3n/4) + O(n) Implies T(n) = O(n)! How do we find such a pivot? Randomly? In fact works! Analysis a little bit later. Can we choose pivot deterministically? Chandra & Manoj (UIUC) CS Fall / 32

41 Divide and Conquer Approach A game of medians Idea 1 Break input A into many subarrays: L 1,... L k. 2 Find median m i in each subarray L i. 3 Find the median x of the medians m 1,..., m k. 4 Intuition: The median x should be close to being a good median of all the numbers in A. 5 Use x as pivot in previous algorithm. Chandra & Manoj (UIUC) CS Fall / 32

42 Example Chandra & Manoj (UIUC) CS Fall / 32

43 Choosing the pivot A clash of medians 1 Partition array A into n/5 lists of 5 items each. L 1 = {A[1], A[2],..., A[5]}, L 2 = {A[6],..., A[10]},..., L i = {A[5i + 1],..., A[5i 4]},..., L n/5 = {A[5 n/5 4,..., A[n]}. 2 For each i find median b i of L i using brute-force in O(1) time. Total O(n) time 3 Let B = {b 1, b 2,..., b n/5 } 4 Find median b of B Chandra & Manoj (UIUC) CS Fall / 32

44 Choosing the pivot A clash of medians 1 Partition array A into n/5 lists of 5 items each. L 1 = {A[1], A[2],..., A[5]}, L 2 = {A[6],..., A[10]},..., L i = {A[5i + 1],..., A[5i 4]},..., L n/5 = {A[5 n/5 4,..., A[n]}. 2 For each i find median b i of L i using brute-force in O(1) time. Total O(n) time 3 Let B = {b 1, b 2,..., b n/5 } 4 Find median b of B Lemma Median of B is an approximate median of A. That is, if b is used a pivot to partition A, then A less 7n/ and A greater 7n/ Chandra & Manoj (UIUC) CS Fall / 32

45 Algorithm for Selection A storm of medians select(a, j): Form lists L 1, L 2,..., L n/5 where L i = {A[5i 4],..., A[5i]} Find median b i of each L i using brute-force Find median b of B = {b 1, b 2,..., b n/5 } Partition A into A less and A greater using b as pivot if ( A less ) = j return b else if ( A less ) > j) return select(a less, j) else return select(a greater, j A less ) Chandra & Manoj (UIUC) CS Fall / 32

46 Algorithm for Selection A storm of medians select(a, j): Form lists L 1, L 2,..., L n/5 where L i = {A[5i 4],..., A[5i]} Find median b i of each L i using brute-force Find median b of B = {b 1, b 2,..., b n/5 } Partition A into A less and A greater using b as pivot if ( A less ) = j return b else if ( A less ) > j) return select(a less, j) else return select(a greater, j A less ) How do we find median of B? Chandra & Manoj (UIUC) CS Fall / 32

47 Algorithm for Selection A storm of medians select(a, j): Form lists L 1, L 2,..., L n/5 where L i = {A[5i 4],..., A[5i]} Find median b i of each L i using brute-force Find median b of B = {b 1, b 2,..., b n/5 } Partition A into A less and A greater using b as pivot if ( A less ) = j return b else if ( A less ) > j) return select(a less, j) else return select(a greater, j A less ) How do we find median of B? Recursively! Chandra & Manoj (UIUC) CS Fall / 32

48 Algorithm for Selection A storm of medians select(a, j): Form lists L 1, L 2,..., L n/5 where L i = {A[5i 4],..., A[5i]} Find median b i of each L i using brute-force B = [b 1, b 2,..., b n/5 ] b = select(b, n/10 ) Partition A into A less and A greater using b as pivot if ( A less ) = j return b else if ( A less ) > j) return select(a less, j) else return select(a greater, j A less ) Chandra & Manoj (UIUC) CS Fall / 32

49 Running time of deterministic median selection A dance with recurrences T(n) = T( n/5 ) + max{t( A less ), T( A greater ) } + O(n) Chandra & Manoj (UIUC) CS Fall / 32

50 Running time of deterministic median selection A dance with recurrences T(n) = T( n/5 ) + max{t( A less ), T( A greater ) } + O(n) From Lemma, T(n) T( n/5 ) + T( 7n/ ) + O(n) and T(n) = O(1) n < 10 Chandra & Manoj (UIUC) CS Fall / 32

51 Running time of deterministic median selection A dance with recurrences T(n) = T( n/5 ) + max{t( A less ), T( A greater ) } + O(n) From Lemma, T(n) T( n/5 ) + T( 7n/ ) + O(n) and T(n) = O(1) n < 10 Exercise: show that T(n) = O(n) Chandra & Manoj (UIUC) CS Fall / 32

52 Median of Medians: Proof of Lemma Proposition There are at least 3n/10 6 elements greater than the median of medians b. Figure : Shaded elements are all greater than b Chandra & Manoj (UIUC) CS Fall / 32

53 Median of Medians: Proof of Lemma Proposition There are at least 3n/10 6 elements greater than the median of medians b. Figure : Shaded elements are all greater than b Proof. At least half of the n/5 groups have at least 3 elements larger than b, except for last group and the group containing b. Hence number of elements greater than b is: 3( (1/2) n/5 2) 3n/10 6 Chandra & Manoj (UIUC) CS Fall / 32

54 Median of Medians: Proof of Lemma Proposition There are at least 3n/10 6 elements greater than the median of medians b. Corollary A less 7n/ Via symmetric argument, Corollary A greater 7n/ Chandra & Manoj (UIUC) CS Fall / 32

55 Questions to ponder 1 Why did we choose lists of size 5? Will lists of size 3 work? 2 Write a recurrence to analyze the algorithm s running time if we choose a list of size k. Chandra & Manoj (UIUC) CS Fall / 32

56 Median of Medians Algorithm Due to: M. Blum, R. Floyd, D. Knuth, V. Pratt, R. Rivest, and R. Tarjan. Time bounds for selection. Journal of Computer System Sciences (JCSS), Chandra & Manoj (UIUC) CS Fall / 32

57 Median of Medians Algorithm Due to: M. Blum, R. Floyd, D. Knuth, V. Pratt, R. Rivest, and R. Tarjan. Time bounds for selection. Journal of Computer System Sciences (JCSS), How many Turing Award winners in the author list? Chandra & Manoj (UIUC) CS Fall / 32

58 Median of Medians Algorithm Due to: M. Blum, R. Floyd, D. Knuth, V. Pratt, R. Rivest, and R. Tarjan. Time bounds for selection. Journal of Computer System Sciences (JCSS), How many Turing Award winners in the author list? All except Vaughn Pratt! Chandra & Manoj (UIUC) CS Fall / 32

59 Takeaway Points 1 Recursion tree method and guess and verify are the most reliable methods to analyze recursions in algorithms. 2 Recursive algorithms naturally lead to recurrences. 3 Some times one can look for certain type of recursive algorithms (reverse engineering) by understanding recurrences and their behavior. Chandra & Manoj (UIUC) CS Fall / 32