Greedy. Outline CS141. Stefano Lonardi, UCR 1. Activity selection Fractional knapsack Huffman encoding Later:

Transcription

1 October 5, 017 Greedy Chapters 5 of Dasgupta et al. 1 Activity selection Fractional knapsack Huffman encoding Later: Outline Dijkstra (single source shortest path) Prim and Kruskal (minimum spanning tree) Stefano Lonardi, UCR 1

2 Optimization problems A class of problems in which we are asked to find a set (or a sequence) of items that satisfy some constraints and simultaneously optimize (i.e., maximize or minimize) some objective function 3 Greedy method Typically applied to optimization problems, that is, problems that involve searching through a set of configurations to find one that minimizes/maximizes an objective function defined on these configuration Greedy strategy: at each step of the optimization procedure, choose the configuration which seems the best between all of those possible 4 Stefano Lonardi, UCR

3 Searching for the global minimum objective function a series of locally optimal moves may NOT lead to the global optimal configurations 5 Greedy method There are problem for which the globally optimal solution can be found by making a series of locally optimal (greedy) choices Make whatever choice seems best at the moment and then solve the sub-problem arising after the choice is made The choice made by a greedy algorithm may depend on choices so far, but it cannot depend on any future choices or on the solutions to sub-problems The greedy strategy does not always lead to the global optimal solution 6 Stefano Lonardi, UCR 3

4 Searching for the global minimum objective function a series of locally optimal moves will always lead to the global optimal configurations 7 Elements of greedy strategy Two ingredients that are exhibited by most problems that lend themselves to a greedy strategy Greedy-choice property: a globally optimal solution can be reached by making a locally optimal choice Optimal substructure: optimal solution to the problem results from optimal solutions to subproblems 8 Stefano Lonardi, UCR 4

5 Activity selection (aka, task scheduling ) 9 Activity Selection Input: A set of activities S = {a 1,, a n } Each activity has start time and a finish time a i =(s i, f i ) Two activities are conflicting if and only if their interval overlap Output: a maximum-size subset of nonconflicting activities 10 Stefano Lonardi, UCR 5

6 Activity Selection Here are a set of start and finish times i s i f i What is the maximum number of activities that can be completed? {a 3, a 9, a 11 } can be completed But so can {a 1, a 4, a 8, a 11 } which is a larger set But it is not unique, consider {a, a 4, a 9, a 11 } Stefano Lonardi, UCR 6

7 Stefano Lonardi, UCR 7

8 Greedy Strategies 1. Longest first. Shortest first 3. Early start first 4. Early finish first 5. None of the above 16 Stefano Lonardi, UCR 8

9 Early Finish Greedy strategy Sort the activities by finish time Schedule first activity (activity 1) Remove all activities that conflict with 1 Recurse on the remaining activities Stefano Lonardi, UCR 9

10 Stefano Lonardi, UCR 10

11 Stefano Lonardi, UCR 11

12 Stefano Lonardi, UCR 1

13 Activity selection in Python def greedy_activity_selection(a): A.sort(key=itemgetter(1)) result = [A[0]] i = 0 for j in range(1,len(a)): if A[j][0] >= A[i][1]: result.append(a[j]) i = j return result Remark: sort A by finish time Remark: first activity in the solution Remark: start[j] >= finish[i] Time complexity? O(n log n) to sort, the rest is linear. 5 Greedy Goal: build a solution in steps, never making a mistake --- maintain the invariant that the partial solution so far is always extendible to an optimal solution. Choosing activity 1 for the first job maximizes the set of remaining (possible, non-conflicting) jobs. 6 Stefano Lonardi, UCR 13

14 Correctness (optimality) Greedy choice property: First choice is consistent with some opt l soln Optimal substructure property: After this first choice, to solve the entire problem optimally, it is enough to solve the remaining sub-problem optimally. 7 Greedy-Choice Property We want to show there is an optimal solution that begins with a greedy choice (i.e., with the first activity, which has the earliest finish time) 8 Stefano Lonardi, UCR 14

15 Suppose Greedy-Choice Property A Í S is an optimal solution Order the activities in A by finish time Let k be the first activity in A If k =1, the schedule A begins with a greedy choice If k 1, show that there is another optimal solution B that begins with the greedy choice (activity 1) Let B = (A {k}) {1} Activities in B are non-conflicting because activities in A are non-conflicting, k is the first activity to finish and f 1 f k B has the same number of activities as A thus, B is optimal 9 Optimal Substructure After the greedy choice of the first activity, the problem reduces to finding an optimal solution for the activity-selection problem over those activities in S that are compatible with the first activity. Formally: Remaining sub-problem is S = { i in S: s i f 1 }. A is an optimal solution for S if and only if A U {1} is an optimal solution for S. 30 Stefano Lonardi, UCR 15

16 Optimal Substructure Remaining sub-problem is S = { i in S: s i f 1 }. Claim. A is an optimal solution for S if and only if A U {1} is an optimal solution for S. Proof. (=> direction) Let A be any optimal solution for S. Then A U {1} is a solution for S. (why?) If it is not optimal for S, then (by greedy choice) there is a larger solution B U {1} for S. But then B is a solution for S (why?), and B is larger than A. (We leave the <= direction as an exercise.) QED 31 Proof. Claim. Greedy is optimal greedy(s) = {1} U greedy(s ) - defn of greedy = {1} U opt(s ) - induction on S = opt(s) - optimal substructure base case: greedy({}) = {} = opt({}) 3 Stefano Lonardi, UCR 16

17 Fractional Knapsack 33 Fractional Knapsack Given a set S of n items, such that each item i has a positive benefit b i and a positive weight w i ; the size of the knapsack W The problem is to find the amount x i of each item i which maximizes the total benefit å i b i (x i / w i ) under the condition that 0 x i w i and W å x i i 34 Stefano Lonardi, UCR 17

18 Fractional Knapsack - Example W=50 Item 1 w 1 =10 10 Item w =0 0 Item 3 w 3 = $ $ $60 b 1 =$60 b =$100 b 3 =$10 $40 $6/pound $5/pound $4/pound 35 Fractional Knapsack in Python def fractional_knapsack(s, W): v = [] for item in S: value = float(item[1]) / float(item[0]) v.append((value,item[1],item[])) v.sort(key=itemgetter(0)) w, result = 0, [] while w < W: high = v[-1] v.pop() a = min(high[1], W-w) Remark: select and remove the highest value (high) w += a result.append((a,high[])) return result Remark: sort v by value = benefit/weight Remark: a is how much item high we took 36 Stefano Lonardi, UCR 18

19 Fractional Knapsack Time complexity is O(n log n) Fact: Greedy strategy is optimal for the fractional knapsack problem Proof: We will show that the problem has the optimal substructure and the algorithm satisfies the greedy-choice property 37 Greedy Choice property holds Items (sorted by b i /w i ) 1 3 j n Optimal solution: x 1 x x 3 x j x n While x 1 < min(w 1, W): For some small d > 0, increase x 1 by d, decrease some x j by d. Increase in total benefit, d (b 1 / w 1 - b j/ / w j ), is non-negative. Stop when x 1 = min(w 1, W). Yields equally good solution with x 1 = min(w 1, W). 38 Stefano Lonardi, UCR 19

20 Optimal substructure property Let x 1 = min(w 1,W) be the greedy choice for the first step. (S, W) is the original problem, (S,W ) is the sub-problem, where S ={,3,,n}, W =W- x 1 items: 1 3 n soln for (S,W ): - x x 3 x n soln for (S, W ): x 1??? x, x 3, x n is an optimal solution to (S, W ) if and only if x 1, x, x 3, x n is an optimal solution to (S, W) Proof. Left as an exercise! 39 Huffman codes 40 Stefano Lonardi, UCR 0

21 Data Compression Text files are usually stored by representing each character with an 8-bit ASCII code The ASCII encoding is an example of fixed-length encoding, where each character is represented with the same number of bits In order to reduce the space required to store a text file, we can exploit the fact that some characters are more likely to occur than others 41 Data Compression Variable-length encoding uses binary codes of different lengths for different characters; thus, we can assign fewer bits to frequently used characters, and more bits to rarely used characters Huffman coding (section 5.) 4 Stefano Lonardi, UCR 1

22 File Compression: Example An example text: java encoding: a = 0, j = 11, v = 10 encoded text: (6 bits) How to decode in the case of ambiguity? encoding: a = 0, j = 11, v = 00 encoded text: (6 bits) could be java, or jvv, or jaaaa, or 43 Encoding To prevent ambiguities in decoding, we require that the encoding satisfies the prefix rule: no code is a prefix of another Example a = 0, j = 11, v = 10 satisfies the prefix rule a = 0, j = 11, v= 00 does not satisfy the prefix rule (the code of 'a' is a prefix of the codes of 'v') 44 Stefano Lonardi, UCR

23 Trie We use an encoding trie to satisfy this prefix rule the characters are stored at the external nodes a left child (edge) means 0 a right child (edge) means 1 D B C A R A = 010 B = 11 C = 00 D = 10 R = Example of Decoding encoded text: text: ABRACADABRA (11 bytes=88 bits) D B C A R A = 010 B = 11 C = 00 D = 10 R = Stefano Lonardi, UCR 3

24 Data Compression Problem: We want the encoded text as short as possible Example: ABRACADABRA bits C D B A R 47 Data Compression Example: ABRACADABRA bits A B R C D 48 Stefano Lonardi, UCR 4

25 Optimization problem Given a character c in the alphabet Σ let f(c) be the frequency of c in the file let d T (c) be the depth of c in the tree = the length of the codeword We want to minimize the number of bits required to encode the file, that is å min f( c) dt ( c) T c binary trees with å leaves ÎS 49 Huffman Encoding: Example Step 0 ABRACADABRA character frequency A B R 5 C D 1 1 Step 1 5 A B R C 1 D 1 50 Stefano Lonardi, UCR 5

26 Huffman Encoding: Example Step 5 A 4 B R C 1 D 1 Step A B R C D 51 Huffman Encoding: Example Step A B R C D 11 Step 5 5 A B R C D 5 Stefano Lonardi, UCR 6

27 Final Huffman Trie A B R A C A D A B R A (3 bits) 5 A B R C D 53 Step 0 Another Example ABRACADABRA character frequency A B R 5 C D 1 1 Step 1 5 A B R C 1 D 1 Step 5 A B R 4 C 1 D 1 54 Stefano Lonardi, UCR 7

28 Another Example 5 A B 4 Step 3 R C 1 D A Step 4 B 4 R C 1 D 1 55 Another Example 5 6 Step 5 A B 4 R C 1 D Step 6 A B 4 R C 1 D 1 56 Stefano Lonardi, UCR 8

29 Final Trie A B R A C A D A B R A (3 bits) 11 5 A B R C 1 D 1 57 Priority queue Use a priority queue for storing the nodes Priority queue is a queue ordered by priority (heap) For our application, priority = frequency If there are k elements in the queue: Extracting the lowest priority is O(log k) Inserting takes O(log k) 58 Stefano Lonardi, UCR 9

30 Huffman algorithm in Python def makehufftree(t): heapq.heapify(t) Remark: transforms list t in a heap in linear time while len(t) > 1: L, R = heapq.heappop(t), heapq.heappop(t) parent = (L[0] + R[0], L, R) heapq.heappush(t, parent) return t[0] Remark: returns the tree represented a nested tuple def printhufftree(t, prefix = ''): if len(t) == : print t[1], prefix else: printhufftree(t[1], prefix + '0') printhufftree(t[], prefix + '1') 59 Huffman Algorithm Running time for a text of length n with k distinct characters: O(n + k log k) If we assume k to be a constant (i.e., not a function of n) then the algorithm runs in O(n) time Fact: Using a Huffman encoding trie, the encoded text has minimal length Proof: omitted 60 Stefano Lonardi, UCR 30

31 Greedy method: summary Task scheduling Fractional knapsack Huffman encoding (section 5.) Other greedy algorithms that will be covered later: Prim (section 5.1.5), Kruskal (section 5.1.3) and Dijkstra (section 4.4) 61 Stefano Lonardi, UCR 31