2 Scalable algorithms, Scheduling, and a glance at All Prefix Sum

Transcription

1 CME 323: Distributed Algorithms ad Otimizatio, Srig 2017 htt://staford.edu/~rezab/dao. Istructor: Reza Zadeh, Matroid ad Staford. Lecture 2, 4/5/2017. Scribed by Adreas Satucci. 2 Scalable algorithms, Schedulig, ad a glace at All Prefix Sum 2.1 Tyes of scalig Oe imortat otio is the idea of how much seed-u we ca hoe to achieve give more rocessors. We have three differet tyes of scalability. Strog scalig Let T 1, deote the ru-time o oe rocessor give a iut of size. Suose we have rocessors. We defie the seed u of a arallel algorithm as SeedU(, ) = T 1, T,. Defiitio 2.1 If SeedU(, ) = Θ(), we say that the algorithm is strogly scalable. Last lecture, we saw arallel sum of umbers o rocessors, which is strogly scalable. 1 Weak Scalig There is aother otio of scalability. We sometimes are iterested to cosider the case where for each rocessor we add, we add more data as well. This is actually quite realistic, sice ofte the oly time we ca afford to add more rocessors or machies is whe we are burdeed with more data tha our ifrastructure ca hadle. Defiitio 2.2 If SeedU(,) = T 1, T, = Ω(1), the our algorithm is weakly scalable. The last otio of scalability is the case whe our algorithm is embarrassigly arallel. This is true if our DAG has odes 0-deth, e.g. Figure 1: A Embarrassigly Parallel DAG That is, there is o deedecy betwee our oeratios. It s scalable i the most trivial sese, e.g. fliig as may cois as ossible at the same time. 1 To see this, recall that work is T 1 =, ad deth is T = log 2, so SeedU(, ) = = Θ(). Secifically, +log 2 both ad are assumed to be goig to ifiity, but grows much slower tha does. Hece + log 2 + ad 2 + log 2 =. 1

2 2.2 Schedulig Parallelism is almost etirely about buildig algorithms with low deth, ad clever schedulig. We first motivate why schedulers are imortat. Motivatio So far, we have assumed that rocessors are assiged tasks i a otimal way. However, the rocess of assigig tasks to rocessors is actually a o-trivial roblem i ad of itself. So, suose you have a rogram which is very large ad recursive i ature. At the ed of the day, it s a DAG of comutatios. At ay level i the DAG, there are a certai umber of comutatios which ca be required to execute (at the same time). It s ossible that the umber of comutatios to be more (or less) tha the umber of rocessors you have available to you. Ideally, you wish for all your rocessors to be busy. Deedig o how you schedule the oeratios, you sometimes may ed u with rocessors which are idle ad ot workig. It is the schedulers task to schedule thigs i tadem i such a way that you look ahead a little bit ad miimize the idle time of rocessors. We could do this greedily, i.e. as soo as there is ay comutatio to be doe, we assig it to a rocessor. Or, we could be a little bit clever about it, ad erhas look ahead further to our DAG to see if we ca la more efficietly. Sark has a scheduler. Every distributed comutig set u has a scheduler. Your oeratig system ad hoe s have schedulers. Every comuter has rocesses, ad every comuter rus i arallel. Your comuter might have fifty Chrome tabs oe ad must decide which oe to give riority to i order to otimize erformace of your machie Problem defiitio A imortat roblem i ay arallel or distributed comutig settig is figurig out how to schedule jobs otimally. I.e. a scheduler must be able to assig sequetial comutatio to rocessors or machies i order to miimize the total time ecessary to rocess all jobs. Notatio We assume that the rocessors are idetical (i.e. each job takes the same amout of time to ru o ay of the machies). More formally, we are give rocessors ad a uordered set of jobs with rocessig times J 1,..., J R. Say that the fial schedule for rocessor i is defied by a set of idices of jobs assiged to rocessor i. We call this set S i. The load for rocessor i is therefore, L i = k S i J k. The goal is to miimize the makesa defied as L max = max i {1,...,} L i The simle (greedy) algorithm The ituitio behid the greedy algorithm discussed here is simle: i order to miimize the makesa we do t wat to give a job to a machie that already has a large load. Therefore, we cosider the followig algorithm. Take the jobs oe by oe ad assig each job to the rocessor that has the least load at that time. This algorithm is simle ad is olie. 2

3 1 for each job that comes i (streamig) do 2 Assig job to lowest burdeed machie 3 ed Algorithm 1: Simle scheduler Other variats of schedulig We ote there are may other variats of schedulig. Jobs ca have deedecies, i.e. oe job must fiish before aother job ca start. Here, the roblem is resecified by a comutatioal DAG that is kow before the time of schedulig. Aother variat is that schedulig must hae olie, i.e. jobs come at you i a order where you caot look ito the future. As jobs come i, you have to schedule it, ad you caot go back ad chage the schedule. For a comrehesive treatmet of variats of schedulig, see Hadbook of Schedulig Otimality of the greedy aroach I either of the above cases, where jobs have deedecies or must be scheduled olie, the roblem is NP hard. So, we use aroximatio algorithms. We claim that the simle (greedy, ad olie) algorithm actually has a aroximatio ratio of 2. I other words, the algorithm is i the worstcase 2 times worse tha the otimal, which is fairly good. For this aalysis, we defie the otimal makesa (miimal makesa ossible) to be OPT ad try to comare the outut of the greedy algorithm to this. We also defie L max as above to be the makesa. Claim: Greedy algorithm has a aroximatio ratio of 2. Proof: We first wat to get a hadle (lower boud) o OPT. We kow that the otimal makesa must be at least the sum of the rocessig times for the jobs divided amogst the rocessors, i.e. 3 OPT 1 J i. (1) A secod lower boud is that OPT is at least as large as the time of the logest job, 4 OPT max i J i. (2) Now cosider ruig the greedy algorithm ad idetifyig the rocessor resosible for the makesa of the greedy algorithm (i.e. k = argmax i L i ). Let J t be the load of the last job laced 2 To give a idea of aother variat, cosider the case of distributed comutig, where each machie houses a set of local data, ad shufflig data across the etwork is a bottleeck. We may cosider schedulig jobs to machies such that o data are shuffled. We ll cosider this more i the latter art of the course. This is called locality sesitive schedulig 3 To see this, assume toward cotradictio that OPT is able to schedule jobs such that OPT < 1 Ji. Suose that istead of OPT assigig jobs to rocessors i arallel, we assiged all the work to oe rocessor sequetially. The of course the total time required give by OPT < Ji but this is a cotradictio, sice Ji exactly reresets the amout of work required to rocess all jobs o a sigle rocessor. 4 The reaso for this is simle: the logest job must be scheduled at some oit to be ru sequetially o oe rocessor, at which oit it will require max i J i time to comute. There may be other rocessors which bottleeck our makesa, but we kow that OPT must take at least as log as ay job, ad i articular this holds for the largest job. 3

4 o this rocessor. Before the last job was laced o this rocessor, the load of this rocessor was thus L max - J t. By the defiitio of the greedy algorithm, this rocessor must have also had the least load before the last job was assiged. Therefore, all other rocessors at this time must have had load at least L max J t, i.e. L max J t L i for all i. Hece, summig this iequality over all i, (L max J t ) L i L i = J i (3) I the secod iequality, we assert that although J t the last job laced o the bottleeck rocessor, there may still be other jobs yet to be assiged, hece we have that the sum of total work laced o each machie caot decrease after assigig all jobs. The last equality comes from the fact that the sum of the loads must be equal to the sum of the rocessig times for the jobs. Rearragig the terms i this exressio let s us exress this as: L max 1 J i + J t (4) Now, ote that our greedy algorithm actually has makesa exactly equal to L max, by defiitio. Usig equatios 1 ad 2 alog with the fact that J t max i J i, we get that our greedy aroximatio algorithm has makesa APX = L max OP T + OP T = 2 OP T. (5) This shows that the greedy algorithm rovides us with a schedulig time that is ot more tha 2 times more tha the otimal. What if we could see the future? We ote that if we first sort the jobs i descedig order ad assig larger jobs first, we ca aively get a 3/2 aroximatio. The ituitio is that if we first schedule large jobs, we ca use the smaller jobs to fill i the gas remaiig, i.e. to balace all loads. If we use the same algorithm with a tighter aalysis, we get a 4/3 aroximatio. We ll see later i the course how Sark uses lazy evaluatio for exactly this reaso: by fakig comutatio util the user takes a actio, Sark ca sort jobs ad to obtai a more efficiet scheduler. What s realistic? It may seem that our above assumtio, to be able to look ito the future ad kow which jobs are goig to be scheduled, is quite ureasoable. I reality, we do t eve kow how log each job will take. However, we ofte have a retty good idea (based o historical data or exectatios) how log a articular job will take to ru. Ad further, we may have statistics or exectatios o how may jobs of a articular tye are goig to come i, hece, it may ot be such a urealistic sceario to kow (withi a certai tolerace) the exected amout of time each job will take as well as what jobs might be i the ielie. 2.3 All Prefix Sum Give a list of itegers, we wat to fid the sum of all refixes of the list, i.e. the ruig sum. We are give a iut array A of size elemets log. Our outut is of size + 1 elemets log, ad its first etry is always zero. 4

5 As a examle, suose A = [3, 5, 3, 1, 6], the AllPrefixSum(A) = [0, 3, 8, 11, 12, 18]. This feels like a iheretly sequetial task. The obvious way to do this with a sigle machie is to have a ruig sum ad write itermediary sums as we iterate through the array i liear time. However, this does ot arallelize at all. How ca we arallelize this, so that it has low-deth? We ll take a look at this roblem i more detail ext lecture. For ow, try to come u with your ow algorithm. Refereces [1] Ola Svesso Aroximatio Algorithms. EPFL, Jauary 21, [2] Joseh Y-T. Leug. Hadbook of Schedulig. CRC Press, [3] G. Blelloch ad B. Maggs. Parallel Algorithms. Caregie Mello Uiversity. 5