Randomized algorithms. Inge Li Gørtz

Transcription

1 Randomized algorithms Inge Li Gørtz

2 Randomized algorithms Today What are randomized algorithms? Properties of randomized algorithms Two examples: Median/Select. Quick-sort

3 Randomized Algorithms

4 Randomized Algorithms So far we dealt with deterministic algorithms: Giving the same input to the algorithm repeatedly results in: The same running time. The same output. Randomized algorithm: Can make random choices (using a random number generator)

5 Random number generator A randomized algorithm has access to a random number generator rand(a, b), where a, b are integers, a < b. rand(a, b) returns a number x {a, (a + 1),..., (b 1), b}. Each of the b a + 1 numbers is equally likely. Calling rand(a, b) again results in a new, possibly different number. rand(a, b) has no memory. The current number does not depend on the results of previous calls. Statistically speaking: rand(a, b) generates numbers independently according to uniform distribution.

6 Randomized algorithms The algorithm can make random decisions using rand Running while (rand(0,1) = 0) do print( running ); end while print( stopped ); Julefrokost while (rand(1,4) = rand(1,4)) do have another Christmas beer; end while go home;

7 Properties of randomized algorithms A randomized algorithm might never stop. Example: Find the position of 7 in a three-element array containing the numbers 4, 3 and 7. Find7(A[1 3]) goon := true; while (goon) do i := rand(1,3); if (A[i] = 7) then print(bingo: 7 found at position i); goon := false; end if end while

8 Properties of randomized algorithms A randomized algorithm might find the wrong solution. Example: Find the position of the minimum element in a three-element array. MinOfThree(A[1 3]) i := rand(1,3); j := rand(1,3); return(min(a[i],a[j]); If A = [5,6,4] and i = 2 and j = 1 then the algorithm will wrongly return 5.

9 Properties of randomized algorithms Both examples are not so bad as they look. It is highly unlikely that Find7 runs a long time. It is more likely that MinOfThree returns a correct result than a wrong one.

10 Events An event is something that can happen. An atomic event is not composed of others. Each atomic event x has an associated probability P[x] [0,1]. The sum of the probabilities of all atomic events is 1. X x is atomic event P [x] =1 We consider only finite or countably infinite sets of atomic events.

11 Random variables A random variable is an entity that can assume different values. The values are selected randomly ; i.e., the process is governed by a probability distribution. Examples: The number shown by a die. The distance of two coins dropped on the floor. The running time of a randomized algorithm.

12 Expected values Let X be a random variable with values in {x1, xn}, where xi are numbers. Let pi = P[X = xi] be the probability that X assumes value xi. The expected value (expectation) of X is defined as E[X] = x i p i i=1 The expectation is the theoretical average.

13 Example: a fair die Let X be the random variable number shown by the die. Atomic events: {1, 2, 3, 4, 5, 6} Probabilities: P[1] = = P[6] = 1/6 Composed event: Even = {2, 4, 6} Composed event: ge4 = {4, 5, 6} Expectation: E[X] = ( )/6 = 3.5

14 Example: a loaded die Let X be the random variable number shown by the die. Atomic events: 1, 2, 3, 4, 5, 6 Probabilities: 0.1, 0.1, 0.1, 0.3, 0.2, 0.2 Expectation E[X] = = 4.0

15 Expectation If we repeatedly perform independent trials of an experiment, each of which succeeds with probability p > 0, then the expected number of trails we need to perform until the first succes is 1/p. Linearity of expectation: For two random variables X and Y we have E[X+Y] = E[X] + E[Y].

16 Rules for Probabilities Let A and B be events. If A and B are independent then P[A B] = P[A] P[B] If A and B are disjoint then P[A B] = P[A] + P[B]

17 Independence example A die is tossed twice. Let the random variable X1 be the result of first throw. Let the random variable X2 be the result of second throw. Events: A = (X1 = 2) and B = (X2 = 5) P[A] = 1/6 and P[B] = 1/6 P[A B] = 1/36 = 1/6 1/6 = P[A] P[B]

18 Non-independent example Event A = Even = {2, 4, 6}: P[A] = 1/2 Event B = ge4 = {4, 5, 6}: P[B] = 1/2 A B = {4, 6} P[A B] = 1/3 1/4 = P[A] P[B]

19 Analysis of MinOfThree MinOfThree(A[1 3]) i := rand(1,3); j := rand(1,3); return(min(a[i],a[j]); Running time: O(1). What is the chance that the answer of MinOfThree is correct? 9 possible pairs all with probability 1/9: (1,1), (1,2), (1,3), (2,1), (2,2), (2,3), (3,1), (3,2), (3,3) Assume wlog that A[1] is the minimum: 5 of the 9 pairs contain the index 1 and give the correct minimum. P[correct] = 5/9 > 1/2.

20 Analysis of MinOfThree MinOfThree(A[1 3]) i := rand(1,3); j := rand(1,3); return(min(a[i],a[j]); Running MinOfThree once gives the correct answer with probaility 5/9 and the incorrect one with probability 4/9. Idea: Run MinOfThree many times and pick the smallest value returned. The probability that MinOfThree fails every time in k runs is: To be 99% sure to find the minimum, choose k such that

21 Analysis of Find7 Find7(A[1 3]) goon := true; while (goon) do i := rand(1,3); if (A[i] = 7) then print(bingo: 7 found at position i); goon := false; end if end while Running time is variable. What is the expected running time? Assume wlog that A[1] = 7. If rand(1, 3) = 1 then the while-loop is terminated. P[rand(1,3) = 1] = 1/3 If rand(1, 3) 1 then the while-loop is not terminated. P[rand(1,3) 1] = 2/3 The while-loop is terminated after one iteration with probability P[rand(1,3) = 1] = 1/3.

22 Analysis of Find7 Find7(A[1 3]) goon := true; while (goon) do i := rand(1,3); if (A[i] = 7) then print(bingo: 7 found at position i); goon := false; end if end while The while-loop stops after exactly two iterations if rand(1, 3) 1 in the first iteration. rand(1, 3) = 1 in the second iteration. This happens with probability 2/3 and 1/3, resp. The probability for both to happen is - by independence -

23 Analysis of Find7 The while-loop stops after exactly three iterations if rand(1, 3) 1 in the first two iterations. rand(1, 3) = 1 in the third iteration. This happens with probability 2/3, 2/3 and 1/3, resp. The probability for all three to happen is - by independence - The while-loop stops after exactly k iterations if rand(1, 3) 1 in the first k-1 iterations. rand(1, 3) = 1 in the kth iteration. This happens with probability 2/3, 2/3 and 1/3, resp. The probability for all three to happen is - by independence -

24 Expected running time We have The expected running time of Find7 is Idea: Stop Las Vegas algorithm if it runs much longer than the expected time and restart it.

25 Types of randomized algorithms We have seen two kinds of algorithms: Monte Carlo algorithms: stop after a fixed (polynomial) time and give the correct answer with probability greater 50%. Las Vegas algorithms: have variable running time but always give the correct answer.

26 Median/Select

27 Select Given n numbers S = {a1, a2,, an}. Median: number that is in the middle position if in sorted order. Select(S,k): Return the kth smallest number in S. Min(S) = Select(S,1), Max(S)= Select(S,n), Median = Select(S,n/2). Assume the numbers are distinct. Select(S, k) { } Choose a pivot s S uniformly at random. For each element e in S if e < s put e in S if e > s put e in S if S = k-1 then return s if S k then call Select(S, k) if S < k then call Select(S, k - S - 1)

28 Select Select(S, k) { } Choose a pivot s S uniformly at random. For each element e in S if e < s put e in S if e > s put e in S if S = k-1 then return s if S k then call Select(S, k) if S < k then call Select(S, k - S - 1) Worst case running time: T (n) =cn + c(n 1) + c(n 2) + = (n 2 ). If there is at least an Limit number of bad pivots. " fraction of elements both larger and smaller than s: T (n) = cn +(1 ")cn +(1 ") 2 cn + = 1+(1 ")+(1 ") 2 + cn apple cn/". Intuition: A fairly large fraction of elements are well-centered => random pivot likely to be good.

29 Select Phase j: Size of set at most and at least. Element is central if at least a quarter of the elements in the current call are smaller and at least a quarter are larger. At least half the elements are central. Pivot central => size of set shrinks with by at least a factor 3/4 => current phase ends. Pr[s is central] = 1/2. Expected number of iterations before a central pivot is found = 2 => expected number of iterations in phase j at most 2. X: random variable equal to number of steps taken by algorithm. Xj: expected number of steps in phase j. X = X1+ X2 +. Number of steps in one iteration in phase j is at most cn(3/4) j. E[Xj] = 2cn(3/4) j. Expected running time: E[X] = X j n(3/4) j n(3/4) j+1 E[X j ] apple X j 2cn j 3 =2cn X 4 j j 3 apple 8cn. 4

30 Quicksort

31 Quicksort Given n numbers S = {a1, a2,, an} return the sorted list. Assume the numbers are distinct. Quicksort(A,p,r) { if S 1 return S else Choose a pivot s S uniformly at random. For each element e in S if e < s put e in S if e > s put e in S L = Quicksort(S ) R = Quicksort(S ) } Return the sorted list L s R.

32 Quicksort: Analysis Worst case Quicksort requires Ω(n 2 ) comparisons: if pivot is the smallest element in the list in each recursive call. If pivot aways is the median then T(n) = O(n log n). for i < j: random variable ( 1 if a i and a j compared by algorithm X ij = 0 otherwise X total number of comparisons: 1 X = i=1 j=i+1 X ij Expected number of comparisons: 1 E[X] =E[ X ij ]= i=1 j=i+1 1 i=1 j=i+1 E[X ij ]

33 Quicksort: Analysis Expected number of comparisons: 1 E[X] =E[ X ij ]= i=1 j=i+1 1 i=1 j=i+1 E[X ij ] Since Xij only takes values 0 and 1: E[X ij ]=Pr[X ij = 1] ai and aj compared iff ai or aj is the first pivot chosen from Zij = {ai,,aj}. Pivot chosen independently uniformly at random => all elements from Zij equally likely to be chosen as first pivot from this set. We have Thus Pr[X ij = 1] = 2/(j i + 1) E[X] = 1 i=1 j=i+1 E[X ij ] = 1 i=1 j=i+1 Pr[X ij ] = 1 i=1 j=i+1 2 j i +1 = 1 i=1 i+1 k=2 2 k < 1 i=1 k=1 2 k = 1 i=1 O(log n) = O(n log n)

34 Programming Competition Prize for best three teams. Can count as a mandatory assignment.