Divide and Conquer Dynamic Programming Greedy Algorithm

Size: px

Start display at page:

Download "Divide and Conquer Dynamic Programming Greedy Algorithm"

Giles Miles
5 years ago
Views:

1 C 563 (95.573) LGORTHM NLY ND DGN Coceptios of algorithms Divide ad Coquer Dyamic Programmig Dr. Nejib Zaguia C563-W4 lgorithmic Paradigms Divide-ad-coquer. Break up a problem ito two subproblems, solve each sub-problem idepedetly, ad combie solutio to sub-problems to form solutio to origial problem. Dyamic programmig. Break up a problem ito a series of overlappig sub-problems, ad build up solutios to larger ad larger sub-problems. Greedy. Build up a solutio icremetally, myopically optimizig some local criterio. 2 Dyamic Programmig History Bellma. Pioeered the systematic study of dyamic programmig i the 95s. tymology. Dyamic programmig = plaig over time. ecretary of Defese was hostile to mathematical research. Bellma sought a impressive ame to avoid cofrotatio. "it's impossible to use dyamic i a pejorative sese" "somethig ot eve a Cogressma could object to" Referece: Bellma, R.. ye of the Hurricae, utobiography. 3

2 Dyamic Programmig pplicatios reas. Bioiformatics. Cotrol theory. formatio theory. Operatios research. Computer sciece: theory, graphics,, systems,. ome famous dyamic programmig algorithms. Viterbi for hidde Markov models. Uix diff for comparig two files. mith-waterma for sequece aligmet. Bellma-Ford for shortest path routig i etworks. Cocke-Kasami-Youger for parsig cotext free grammars. 4 Dyamic Programmig Geeral idea: try to avoid computig the solutio of the same sub problems more tha oce, by storig the computed sub solutios. This techique is used for optimizatio problems. Computig is doe from bottom to top. There are usually 4 steps i ay dyamic programmig algorithm: Characterize the structure of the optimal solutio. Defie the optimal solutio recursively Compute the optimal solutio from bottom to top. Combie the solutios for smaller problems stocked i the array to costruct a optimal solutio of the origial problem Dr. Nejib Zaguia C563-W4 5 Dyamic Programmig Problem: Matrix-chai multiplicatio How to compute efficietly the product: M = M M 2 M M i : d i, d i+ M = (...(M * M 2 )* M 3 )... M ) M = (M * M 2 )*(M 3 * M 4 ) *... M )... M = (M * (M 2 *... (M - * M )...) To compute the product of 2 matrices (p,q) ad B(q,r) we eed p*q*r multiplicatios. x (2 x (3 x 4 )) Cost(3 x 4 ) = 5 x x Cost(2 x (3 x 4 )) = 2 x 5 x Cost( x (2 x (3 x 4 ))) = x 2 x Total Cost = 25 = x 2 x 3 x 4 x2 2x5 5x x ( x (2 x 3)) x 4 Coût(2 x 3 ) = 2 x 5 x Coût( x (2 x 3 )) = x 2 x Coût(( x (2 x 3)) x 4 ) = x x Total Cost = 22 Dr. Nejib Zaguia C563-W4 6 2

3 Dyamic Programmig How to fid the most efficiet way to isert the parathesis. direct approach by checkig all possibilities will ot work. M = (M M 2 M i ) (M i+ M i+2 M ) T(i) T() = 2-2 T() = / ( - ) T() (4 / 3/2 ) - T() = T(i) T(-i) i= T(-i) Dr. Nejib Zaguia C563-W4 7 Dyamic Programmig Dyamic programmig approach ) tructure of a optimal solutio f M = (M M 2 M i ) (M i+ M i+2 M ) is a optimal solutio the : (M M 2 M i ) is a optimal solutio for M M i ND (M i+ M i+2 M ) is a optimal solutio for M i+ M For i j,, m[i, j] deotes the optimal solutio for the product M i M i+ M j m[i,j] = (i = j) mi { m[i,k] + m[k +, j] + d i- d k d j } (i < j) i<k<j T() k (T(k) T( k) O()) ( 2 ) Dr. Nejib Zaguia C563-W4 8 Dyamic Programmig Dyamic programmig approach How to compute m[, 6]? m [ i, j ] mi { m[i, k] m[k, j] d d d } i - k j i k < j x x x x x 2 x x x x 3 x x x 4 x x 5 x 6 olutio m[,6] Dr. Nejib Zaguia C563-W4 9 3

4 Dyamic Programmig Matrix-Chai-Order( p, ) { for i = to m[i,i] = for le = 2 to for i = to - le + j = i + le - m[i,j] = 8 for k = i to j- q = m[i,k] + m[k+,j] + d[i-]*d[k]*d[j] if q < m[i,j] the m[i,j] = q s[i,j] = k retur s } Dr. Nejib Zaguia C563-W4 Dyamic Programmig hortest path: Floyd algorithm Let G = (V,) be a coected weighted graph, fid all shortest paths betwee ay two vertices i G. tructure of a optimal solutio: f a shortest path betwee ad B cotais a vertex C the the sub paths betwee ad C ad betwee C ad B are optimal C B 5 3 B 2 5 C D 5 Dr. Nejib Zaguia C563-W4 Dyamic Programmig The algorithm: We maitai a 2 dimesioal array D (iitialized with L). Vertices of the graph are umbered from to. There are iteratios. fter iteratio k, D will cotai the shortest paths betwee ay two vertices (i, j), usig oly vertices from the set {, 2,, k}. t iteratio k ad for every pair of vertices (i, j) we check whether or ot the itroductio of vertex k will give a better shortest path. Phase k: L = D, D,.D = D Case. ddig vertex k does ot give a better solutio: Dk [i,j] = Dk-[i,j] Case 2. ll shortest paths betwee i ad j usig the vertices {, 2,, k} use the vertex k as a itermediary vertex. i Dk [i,j] = Dk-[i,k]+ Dk-[k,j]} Dk[i,j] = mi ( Dk-[i,j], Dk-[i,k] + Dk-[k,j] ) We eed a secod 2 dimesioal array P i order to keep track of the actual shortest path betwee ay two vertices: if D[i,k]+D[k,i] < D[i,j] the D[i,j] = D[i,k]+D[k,i] P[i,j] = k k j Dr. Nejib Zaguia C563-W4 2 4

5 Dyamic Programmig xample: D =L D B D D C 5 Dr. Nejib Zaguia C563-W4 3 Dyamic Programmig Cot D 2 D P D=D Dr. Nejib Zaguia C563-W4 4 Dyamic Programmig Floyd(, L[ ] [ ], P[ ][ ]) array D of size x array P of size x D = L; P= for(k=; k<=; k++) for(i=; i<=; i++) for(j=; j<=; j++) if D[i,k]+D[k,j] < D[i,j] D[i,j] = D[i,k]+D[k,j] P[i, j] = k retur D O( 3 ) Costat Dr. Nejib Zaguia C563-W4 5 5

6 Problem: Makig chage for cets usig the least umber of cois. Greedy algorithm: t each step, retur the largest possible coi xample : Cois: 25c c 5c c Makig chage of 87c: 3 x 25c + x c + 2 x c xample 2: Cois: 25c c 2c 5c c Makig chage of 6c: x 2c + 4 x c (Optimal solutio: x c + x 5c + x c (3 cois)) fuctio chage(c: set {cois}; s: iteger): set with repititios := {solutio} while s > do p := largest coi i C such that its value does ot exceed s; := {p} s := s-p retur Dr. Nejib Zaguia C563-W4 6 greedy algorithm go through a sequece of steps with a set of choices at each step. t makes the choice that looks best at the momet. Greedy algorithms are usually simple ad easy to implemet. Typically used for optimizatio problems. Greedy algorithms do ot always lead to a optimal solutio Geeral cheme of a greedy algorithm: C: set of cadidates := while is ot a solutio ad C x := elemet i C that maximizes the selectio criteria C := C - x if ( {x}) is realizable the := {x} if is a solutio the retur else retur o-solutio Dr. Nejib Zaguia C563-W4 7 Greedy algorithm for a task schedulig problem: miimizig the total waitig time. We have a set of tasks {, 2,, } to be processed sequetially by a sigle processor. ach task has a processig time ti. The goal is to miimize the total waitig time of all tasks (time spet by cliet i i the system: from start to fiishig of task i) i= xample: 3 tasks t = 5 There are 6 possible orders: t 2 = 23: 5 + (5 + ) + ( ) = 38 32: 5 + (5 + 3) + ( ) = 3 t 23: + ( + 5) + ( ) = 43 3 = 3 23: + ( + 3) + ( ) = 4 32: 3 + (3 + 5) + ( ) = 29 32: 3 + (3 + ) + ( ) = 34 Dr. Nejib Zaguia C563-W4 8 6

7 Greedy algorithm: at each step we choose a task, amog the oes left, with a miimum processig time. Theorem. The greedy algorithm always lead to a optimal schedulig Proof. Let = (i,i 2,,i ) a arbitrary permutatio of the tasks {,2,, }. f the tasks are scheduled accordig to the order the the total waitig time is: T() = t i + (t i + t i2 ) + (t i + t i2 + t i3 ) +.. uppose : a... b. k= where a < b ad t ia > t ib f we iverse the positios of a ad b i : b... a. ti ti2...tib...tia. ti Dr. Nejib Zaguia C563-W4 9 = ( - k + ) t ik T(') = ( - a + ) t ib + ( - b + ) t ia + ( - k + ) t ik k = k {a,b} Thus: T() = ( - k + ) t ik k= T(') = ( - a + ) t ib + ( - b + ) t ia + ( - k + ) t ik k = k {a,b} T() - T(') = ( - a + ) (t ia - t ib ) + ( - b + ) (t ib - t ia ) = (b-a) (t ia - t ib ) > Therefore the schedulig is better tha. Thus the greedy algorithm is always optimal, ad ay optimal solutio is ecessarily costructed usig the greedy algorithm. Dr. Nejib Zaguia C563-W4 2 other chedulig problem: uit-time tasks with deadlies ad pealties. set ={, 2,, } of uit-tasks. set of iteger deadlies d, d2,, d, such that di for each i, ad task i is supposed to fiish by time di set of o egative weights g, g2,, g such that a gai gi is icurred if task i is fiished by time di. i gi di The sequeces et, 2, 3 correspods to the same gai. Oly task leads to a gai. Possibles eq: gai ,3 65 2, 6 2,3 25 3, 65 4, 8 4,3 45 Dr. Nejib Zaguia C563-W4 2 7

8 set of tasks is realizable if there exists at least oe orderig of the tasks T that permits to all tasks i T to be executed before the delay. Theorem: set of tasks T is realizable if ad oly if all tasks of T are executed before the delay whe the tasks of T are sorted i icreasig order with respect to delay. Proof: ssume that is realizable with some order: t t2 ti tj tk with d(tj) < d(ti). By exchagig ti et tj i the order of we obviously obtai a realizable order: t t2 tj ti tk Repetitively, we obtai a icreasig order which is realizable. The coverse is obviously true. How do we kow if a set of tasks is realizable? t is eough to test the tasks i oe specific order: icreasig order with respect to delay. Dr. Nejib Zaguia C563-W4 22 : t each step, take the task ot cosidered yet, ad with the largest value gi. f the ew set of tasks is realizable, add that task to the solutio, otherwise disregard the task. ort the set of tasks i decreasig order T with respect to gai := O( log ) repeat Choose the task e T with the largest gai T := T - e f e is realizable := e util T is empty [(i-) + i] = 2 - i=2 Number of steps -. Need (i-) comparisos to add the ith task, i comparisos to verify that if the set is realizable Dr. Nejib Zaguia C563-W4 23 Huffma Code statistical compressio of data, it permits to reduce the legth of the codig of a alphabet. The Huffma code (952) is a optimal variable legth, that is the average legth of the coded text is miimal. We otice a size reductio of the order of 2 to 9%. Dr. Nejib Zaguia C563-W4 24 8

9 Codes with fixed legth: each character is coded with the same umber of bits. For C characters we eed log 2 C bits imple, ad easy to maipulate T B N T B N Dr. Nejib Zaguia C563-W4 25 xample 2 variable legth N T B T B N Dr. Nejib Zaguia C563-W4 26 xample 3 N B T TT = ymbol Code T B N Dr. Nejib Zaguia C563-W4 27 9

10 Char. T B N Frequecy Fixed legth Code Total Bits Variable legth Code Total Bits Dr. Nejib Zaguia C563-W4 28 codig should have the prefix property to avoid the ambiguity: a biary sequece caot be the code of a characters ad i the same time the start of the code of aother character. Prefix codes could be represeted by biary trees. Characters are placed as leafs ad the code is the readig of the uique path from the root to the leaf. Dr. Nejib Zaguia C563-W4 29 = Ou P T B N Prefixe!! ymbole Dr. Nejib Zaguia C563-W4 3 T B N P Code

11 Problem: put list of characters with frequecies. Output The biary tree for the variable legth code Total cost = s l s fs l s = umber of bits of the code of character s f s = frequecy of the character s Dr. Nejib Zaguia C563-W4 3 C 355 lgorithmes Voraces Huffma lgorithm(952) Greedy algorithm for variable legth optimal code The algorithm is of greedy type: costruct a loger path for a character with smaller frequecy. For each character we costruct a tree with a sigle vertex labelled with the frequecy of the character. t each stage of the algorithm, we merge the trees with the smallest labels o their root ito a ew tree with a ew root. The ew root is labelled with the sum of the labels of the old roots. The roots of the old trees become the left ad right child of the ew root. The algorithm eds whe we arrive to a uique tree. Dr. Nejib Zaguia C563-W T P L 9 T L T L 2 T T P T P Dr. Nejib Zaguia C563-W4 33

12 39 T3 T2 P T L T4 T T T L T2 39 T3 P 4 Dr. Nejib Zaguia C563-W T5 T3 T2 P T L 67 T4 T T2 T L 47 T6 T5 T4 T3 T P Dr. Nejib Zaguia C563-W4 35 Code 47 T6 T5 T4 T3 T T2 P T L ymbol Code T P L Dr. Nejib Zaguia C563-W4 36 2

13 T T P P L L Dr. Nejib Zaguia C563-W4 37 efficiet implemetatio: use heaps for vertices «priority list» Huffma (C) = umber of characters i C itializatio of the heap Q, (usig the frequecies as priority) for i i..- do z = ew vertex o tree x = xtract-miimum (Q) y = xtract-miimum (Q) left-child z = x log () right=child z = y f[z] = f[x] + f[y] sert (Q, z) ed for Retur tree O() Total Complexity: O( log) Dr. Nejib Zaguia C563-W4 38 other example: Dyamic Programmig Logest commo subsequece problem equeces X =<x, x2,..., xs> ad Y=<y, y2,..., yt> Z =<z, z2,..., zk> is a subsequece of X if there exists a strictly icreasig sequece <i(), i(2),..., i(k)> of idices of X such that for all j, x i(j) = z j <, C, Z, K> is a subsequece of <C,, F, C, C, Z, K, Y> with idex sequece <2, 4, 6, 7> or <2, 5, 6, 7> Z is a commo subsequece of X ad Y if Z is a subsequece of both X ad Y. X=<, B,C, B, D,, B> Y=<B, D, C,, B, > COMMON UBQUNC (C) B,C, logest C (LC) B, C, B, 2/3/25 Dr. Nejib Zaguia C563-W4 39 3

14 other example: Dyamic Programmig Brute force: try all commo subsequece takes expoetial time X =<x, x2,..., xi> Xi is the i-th prefix of X Theorem: Let Z=<z, z 2,..., z k > be ay LC of X ad Y. () f x s = y t the x k =x s =y t ad Z k- is a LC of X s- ad Y t- (2) f x s y t the z k x s implies that Z is a LC of X s- ad Y (3) f x s y t the z k y t implies that Z is a LC of X ad Y t- Dr. Nejib Zaguia C563-W4 4 other example: Dyamic Programmig Proof. () if z k x s, the we could apped x s =y t, to Z ad gets a C of legth k+. f there were a C W of X s- ad Y t- with legth k, the we could apped x s =y t to W ad get a C of X ad Y of legth k+. Thus, a C of X s- ad Y t- with legth k is a LC i.e. Zk- is a LC of X s- ad Y t- X q Xs- q Y q Y q Z q W q (2) if z k x s, the that Z is a C of X s- ad Y. f there were a C W of X s- ad Y with legth greater tha k, the W would also be a C of X s ad Y. X q Xs- q Y Z (3) ame argumet as (2) r Y W Dr. Nejib Zaguia C563-W4 4 other example: Dyamic Programmig Recursive solutio for fidig LC of X ad Y f x s =y t the fid a LC of X s- ad Y t-, ad the apped x s =y t to this LC f x s y t the solve two problems: () Fid a LC of X s- ad Y; (2) fid a LC of X ad Y t- ; whichever of these two LC s is loger, that s a LC of X ad Y. c[i, j]=legth of a LC of the sequeces X i ad Y j whe i= or j= the c[i, j] = whe i>, j>, ad x i =y j the c[i, j]= c[i-, j-]+ whe i>, j> ad x i y j the c[i, j] = max {c[i, j-], c[i-, j]} Complexity: O(st) Dr. Nejib Zaguia C563-W4 42 4

15 other example: Dyamic Programmig LC-Legth(X, Y) set c[i, ] s ad c[, j] s to for i= to s for j= to t if x i =x j the c[i, j] = c[i-, j-]+ else c[i, j]=max{ c[i, j-], c[i-, j]} Dr. Nejib Zaguia C563-W4 43 Greedy vs. Dyamic Programmig - kapsack problem thief robbig a store fids items i-th item has weight w[i] ad is worth v[i] dollars (w[i] ad v[i] are itegers). f the thief ca carry at most L weight i his kapsak, what items should he take to make the most profitable heist? Fractioal kapsack problem: same as above, except that the thief ca take fractios of items (e.g. a item might be gold dust) Dr. Nejib Zaguia C563-W4 44 Greedy vs. Dyamic Programmig Optimal substructures property: Cosider the most valuable load that weights L pouds i - problem, if we remove item j from this load, the remaiig load must be the most valuable load weightig at most L=w[j] that ca be take from the - origial items excludig j. fractioal problem, if we remove w pouds of item j, the remaiig load must be the most valuable load weightig at L=w that ca be take from the - origial items plus w[j] w pouds of item j. Dr. Nejib Zaguia C563-W4 45 5

16 Greedy vs. Dyamic Programmig greedy solutio for the fractioal problem Compute the v[i]/w[i] for each item i. ort i decreasig order the items accordig to the values of v[i]/w[i] For items i= to take as much of item as there while ot exceedig weight limit L Ruig time: O( log ) The same greedy algorithm does ot work for the - kapsack problem Dr. Nejib Zaguia C563-W4 46 tems Greedy vs. Dyamic Programmig Weight Gai tem 6 tem 2 tem 3 2 kapsack Greedy: 6 Optimal: 22 Take the most expesive does ot work: fails for L=3 Dyamic programmig solutio for - problem: ort items by icreasig weight. Let i be the highest-umbered item i a optimal solutio for a L kilo kapsack ad items... The =-{i} must be a optimal solutio for items..i- ad a L-w[i] kilos kapsacks. The value of solutio is v[i] plus the value of solutio. Let c[i, wj]=the value of the solutio for items..i ad maximum weight w whe i= or w= the c[i, w] = whe w[i] >w, the c[i, w]= c[i-, w] whe i> ad w w[i], the c[i, w] = max {v[i]+c[i-, w-w[i]], c[i-, w]} 2 Dr. Nejib Zaguia C563-W Greedy vs. Dyamic Programmig Dyamic --Kapsack: for w= to L c[, w]= for i= to c[i, ]= for w= to L if w[i] w the if v[i] + c[i-, w-w[i]] > c[i-, w] the c[i, w] = v[i] + c[i-, w-w[i]] else c[i, w]=c[i-, w] else c[i, w] = c[i-, w] Complexity: O(L) Dr. Nejib Zaguia C563-W4 48 6

Matrix Multiplication. Data Structures and Algorithms Andrei Bulatov

Matrix Multiplication. Data Structures and Algorithms Andrei Bulatov Matrix Multiplicatio Data Structures ad Algorithms Adrei Bulatov Algorithms Matrix Multiplicatio 7- Matrix Multiplicatio Matrix multiplicatio. Give two -by- matrices A ad B, compute A B. k kj ik ij b a