GrenchMark : a framework for analyzing, testing, and comparing grids Iosup, A.; Epema, D.H.J.

Transcription

1 GrnMr : rmwor or nlyzn, tstn, n omprn rs Iosup, A.; Epm, D.H.J. Puls n: Prons o t 6t Intrntonl Symposum on Clustr Computn n t Gr (CCGr 26, Snpor, My 16-19, 26) DOI: 1.119/CCGRID Puls: 1/1/26 Doumnt Vrson Pulsr s PDF, lso nown s Vrson o Ror (nlus nl p, ssu n volum numrs) Pls t oumnt vrson o ts pulton: A sumtt mnusrpt s t utor's vrson o t rtl upon sumsson n or pr-rvw. Tr n mportnt rns twn t sumtt vrson n t ol puls vrson o ror. Popl ntrst n t rsr r vs to ontt t utor or t nl vrson o t pulton, or vst t DOI to t pulsr's wst. T nl utor vrson n t lly proo r vrsons o t pulton tr pr rvw. T nl puls vrson turs t nl lyout o t ppr nlun t volum, ssu n p numrs. Ln to pulton Ctton or puls vrson (APA): Iosup, A., & Epm, D. H. J. (26). GrnMr : rmwor or nlyzn, tstn, n omprn rs. In Prons o t 6t Intrntonl Symposum on Clustr Computn n t Gr (CCGr 26, Snpor, My 16-19, 26) (pp ). IEEE Computr Soty Prss. DOI: 1.119/CCGRID Gnrl rts Copyrt n morl rts or t pultons m ssl n t pul portl r rtn y t utors n/or otr opyrt ownrs n t s onton o ssn pultons tt usrs rons n y t ll rqurmnts ssot wt ts rts. Usrs my ownlo n prnt on opy o ny pulton rom t pul portl or t purpos o prvt stuy or rsr. You my not urtr strut t mtrl or us t or ny prot-mn tvty or ommrl n You my rly strut t URL ntyn t pulton n t pul portl? T own poly I you lv tt ts oumnt rs opyrt pls ontt us provn tls, n w wll rmov ss to t wor mmtly n nvstt your lm. Downlo t: 22. Nov. 217

2 GRENCHMARK: A Frmwor or Anlyzn, Tstn, n Comprn Grs Alxnru Iosup n D Epm Dlt Unvrsty o Tnoloy, Fulty o Eltrl Ennrn, Mtmts, n Computr Sn Astrt Gr omputn s omn t nturl wy to rt n sr lr sts o tronous rsours. Wt t nrstrutur omn ry or t lln, urrnt r vlopmnt n ptn n on provn tt rs rlly support rl ppltons, n on rtn qut nmrs to qunty ts support. Howvr, r ppltons r ust nnn to mr, n trtonl nmrs v yt to prov rprsnttv n r nvronmnts. To rss ts n-n- prolm, w propos ml-wy ppro: rt n run syntt r worlos omprsn ppltons rprsnttv or toy s rs. For ts purpos, w v sn n mplmnt GRENCHMARK, rmwor or syntt worlo nrton n sumsson. T rmwor rtly ltts syntt worlo moln, oms wt ovr 35 syntt n rl ppltons, n s xtnsl n lxl. W sow ow t rmwor n us or r systm nlyss, untonlty tstn n r nvronmnts, n or omprn rnt r sttns, n prsnt t rsults otn wt GRENCHMARK n our mult-lustr r, t DAS. 1 Introuton In t lon trm, Gr omputn systms (Grs) m t omn t stnr wy o srn tronous rsours, n o rtn tm nto vrtul pltorms, to us y multpl ornztons n npnnt usrs l. Wt t r nrstrutur strtn to mt t rqurmnts o su n mtous ol [3], t urrnt voluton o rs ns on provn tt t n run rl ppltons, rom trtonl squntl n prlll ppltons to nw, r-only, ppltons. As onsqun, tr s lr n or nrtn n runnn worlos omprsn r ppltons or monstrton n tstn purposs. In ts ppr w prsnt GRENCHMARK, rmwor or syntt worlo nrton n sumsson. Wt GRENCHMARK, w try to n ommon roun or r prormn nlyss n, n t lvor o t Prlll Worlos Arv 1, w or t prormn-ornt Gr 1 T Prlll Worlo Arvs ms vrous worlo trs rom rl prlll prouton nvronmnts vll t ttp:// P.O. Box 531, 26 GA Dlt, T Ntrlns Eml: {A.Iosup,D.H.J.Epm}@w.tult.nl ommunty tool tt n lp to rn totr Gr prormn vluton ppros, towrs t ol o uln stnr Gr nmrs. Our mn ontrutons r: Moln n sltn st o rprsnttv rl n syntt r ppltons (Ston 2), nlun ppltons tt rqur o-lloton (Ston 2.2); A systmt ppro to n st o tools or nrtn syntt r worlos or nlyzn, tstn, n omprn ommon r sttns (Ston 3); An xprmntl vlton o our ppro (Ston 4). In our sttn, w us mult-lustr nvronmnt, t DAS [2], t KOALA 2 o-llotn r sulr [11], n t rl ppltons nlu n t Is 3 Jv-s Gr prormmn nvronmnt [13]. 2 A Mol or Syntt Gr Worlos Ts ston prsnts mol or syntt r worlos. 2.1 Gr st n mn mol W ssum r systms omprsn svrl omputn sts. Sts r prov n mntn y nvuls or nsttutons, n r t to r us. Sts my v rnt us pols,.., on st my t 1% rsours or runnn ts lol usrs os, n only sr t rsours to t lol r ommunty no su os xst, wl, t t otr xtrm, notr st my or ll ts rsours t ny tm to nyoy lonn to t lol r ommunty, wtout srmntn twn lol n rmot usrs. E st ontns svrl omputn rsours,.., st s lustr o rsours. Rsours n omputtonl n stor rsours or ot t t sm tm. On st, tr s only on twy ( mn us s n ntry pont to t systm, rom w os n lun n ls n trnsrr to n rom t lustr). 2 KOALA s vlop t TU Dlt, NL; mor normton out KOALA s vll t ttp:// 3 Is s vlop t VU Amstrm, NL, n s rly vll rom ttp:// Prons o t Sxt IEEE Intrntonl Symposum on Clustr Computn n t Gr (CCGRID'6)

3 2.2 Gr ppltons mol In our mol, w onsr two typs o ppltons tt n run n rs, n my nlu n syntt r worlos. Untry ppltons Ts tory nlus snl, untry, ppltons. At most t o prormmn mol must tn nto ount wn runnn n rs (.., lunn nm srvr or lunn n Is o). Typl xmpls nlu squntl n prlll (.., MPI, Jv RMI, Is) ppltons. Compost ppltons Ts tory nlus ppltons ompos o svrl untry or ompost ppltons. T r sulr ns to t nto ount ssus l ts ntr-pnns, vn rsrvton n xtn ult-tolrn, ss t omponnts o prormmn mol. Typl xmpls nlu s o tss, ns o tss, DAG-s ppltons, n ppltons s on nr rps (s Fur 1 or smpl ompost ppltons nrt wt GRENCHMARK). W lso onsr n our mol t noton o olloton: t smultnous lloton o rsours lot n rnt r sts to snl ppltons w onsst o multpl omponnts. T omponnts o o-llot os usully ntrt wt otr (.., onsr t s o ttly-oupl prlll pplton w ppns to run on rsours lot n svrl r sts). Untry n ompost os n ot o-llot. 2.3 Gr worlos mol W ru tt mny o t potntl Gr usrs r urrnt HPC usrs, or r urrntly runnn lr ts o os. For mny worn nvronmnts, n splly or t DAS, t mount o squntl os s snnt or vn omnnt. Tror, untry ppltons wt squntl or prlll strutur, n ompost ppltons wt o tss strutur r to prrr wn rtn r worlos. T worlo strutur s ult usn wll-nown sttstl strutons or moln rsour rqurmnts, wt vlus xtrt rom rl trs or slt rom sts prn y t worlo snr. Jos rrv ynmlly n t systm, n t ntr-rrvl tm o t morty o os n mol wt sttstl struton. Jos rrvl n lso ursty, tt s, mny os my rrv n vry sort tm ntrvl. 3 T GRENCHMARK rmwor Ts ston prsnts t GRENCHMARK rmwor. 3.1 Ovrvw GRENCHMARK s syntt r worlo nrtor n sumttr. It s xtnsl, n tt t llows nw typs o r ppltons to nlu n t worlo nrton, prmtrzl, s t llows t usr to prmtrz t worlos nrton n sumsson, n portl, s ts rrn mplmntton s wrttn n Pyton. T worlo nrtor s s on t onpts o unt nrtors n o o srpton ls (JDF) prntrs. T unt nrtors prou tl srptons on runnn st o ppltons (worlo unt), orn to t worlo srpton prov y t usr. In prnpl, tr s on unt or support pplton typ. T prntrs t t nrt worlo unts n rt o srpton ls sutl or r sumsson. In ts wy, multpl unt nrtors n oupl to prou worlo tt n sumtt to ny r rsour mnr, s lon s t rsour mnr supports tt typ o ppltons. T r ppltons urrntly support y GRENCH- MARK r squntl os, os w us MPI, n Is os. GRENCHMARK nlus st o ovr 35 syntt n rl ppltons. W v mplmnt our syntt ppltons: ssr, squntl pplton wt prmtrzl omputton n mmory rqurmnts, ssro, squntl pplton wt prmtrzl omputton n I/O rqurmnts, smp1, n MPI pplton wt prmtrzl omputton, ommunton, mmory, n I/O rqurmnts, n sw, n pplton mplmntn t o mol us n t Stnr Worlos Formt, rom t Prlll Worlos Arvs (PWA). W us ll t rl ppltons n som o t syntt ppltons nlu n t ult Is struton p. T rson s trol: t Is ppltons losly rsml or r rl-l prlll ppltons, ty v n provn to run on vrty o r sttns, n ty v n torouly sr 4. T Is ppltons om rom t rs o pysl smultons, prlll rnrn, omputtonl mtmts, stt sp sr, onormts, t omprsson, r mtos, n optmzton. Currntly, GRENCH- MARK n sumt os to KOALA, Glous GRAM, n Conor. T worlo sumttr nrts tl rports o t sumsson pross. T rports nlu ll o sumsson ommns, t turnroun tm o o, nlun t r ovr, t totl turnroun tm o t worlo, n vrous sttstl normton. 3.2 Rplyn trs wt GRENCHMARK GRENCHMARK n rply trs rom vrous prouton nvronmnts. Frst, t usr n lo trs n t 4 For t omplt lst o pultons rlt to Is ppltons, pls vst ttp:// Prons o t Sxt IEEE Intrntonl Symposum on Clustr Computn n t Gr (CCGRID'6)

4 Lvl Lvl Lvl Lvl 1 Lvl 2 Lvl 3 Lvl 4 Lvl 5 Lvl 6 Lvl 7 Lvl 8 Lvl 9 #36 #42 #27 #37 #33 #41 #48 #34 #44 #31 #39 #6 #16 #21 #24 #28 #46 #47 #5 #1 #13 #22 #25 #3 #38 #5 #12 #19 #26 #32 #4 #9 #1 #29 #43 #49 # #3 #15 #35 #45 #4 #14 #2 #2 #17 #23 #8 #11 #7 #18 Lvl Lvl 1 Lvl 2 Lvl 3 Lvl 4 Lvl 5 Lvl 6 Lvl 7 Lvl 8 Lvl 9 Lvl 1 Lvl 11 Lvl 12 Lvl 13 Lvl 14 Lvl 15 Lvl 16 #334 #65 #66 #87 #75 #88 #71 #79 #76 #83 #327 #89 #67 #68 #326 #7 #74 #84 #9 #72 #325 #85 #91 #77 #92 #78 #8 #69 #86 #81 # #82 #73 #56 #49 #29 #51 #53 #28 #27 #55 #58 #96 #57 #97 #98 #52 #5 #54 #35 #36 #172 #171 #37 #38 #42 #17 #39 #256 #283 #34 #1 #247 #99 #11 #294 #258 #4 #31 #118 #33 #112 #24 #115 #267 #41 #122 #249 #312 #134 #285 #33 #292 #298 #265 #128 #262 #13 #25 #17 #268 #125 #31 #286 #295 #137 #34 #259 #244 #316 #289 #37 #123 #241 #129 #271 #135 #276 #16 #253 #277 #266 #131 #248 #121 #239 #257 #11 #293 #275 #18 #127 #284 #9 #12 #32 #133 #242 #157 #141 #159 #1 #114 #139 #21 #113 #148 #19 #15 #15 #111 #145 #14 #163 #15 #142 #46 #151 #47 #16 #48 #3 #32 #154 #31 #14 #166 #1 #331 #3 #149 #319 #2 #63 #14 #158 #332 #22 #62 #147 #11 #333 #64 #12 #24 #143 #144 #4 #146 #161 #2 #153 #194 #178 #27 #176 #196 #185 #17 #187 #182 #2 #179 #188 #197 #191 #177 #173 #155 #156 #186 #328 #195 #23 #184 #329 #18 #33 #313 #28 #26 #21 #211 #212 #174 #18 #175 #16 #23 #162 #152 #164 #165 #124 #238 #181 #25 #183 #13 #19 #198 #274 #19 #192 #199 #193 #189 #21 #22 #296 #32 #311 #314 #246 #335 #321 #26 #126 #243 #252 #119 #245 #269 #272 #13 #254 #35 #132 #279 #261 #287 #255 #38 #138 #29 #299 #136 #263 #36 #288 #297 #281 #117 #315 #116 #317 #273 #251 #264 #3 #282 #39 #278 #12 #291 #318 #25 #323 #94 #168 #213 #6 #234 #229 #224 #44 #23 #235 #223 #226 #232 #28 #228 #236 #7 #222 #22 #5 #221 #231 #214 #215 #8 #225 #227 #233 #6 #216 #218 #217 #29 #237 #219 #45 #27 #43 #61 #95 #169 #26 #324 #59 #93 #167 #24 #322 #1 #2 #3 #4 #5 #6 #7 #8 #9 #1 #11 #1 #2 #3 #4 #5 #6 () () () () Fur 1. Compost ppltons nrt wt GRENCHMARK: () rnomly nrt o os; () rnomly nrt n o os; () DAG-s worlow nrt wt rnom mto rom t Stnr Ts Grp St (ttp:// () DAGs worlow nrt rom t SPEC nmr pppp, nlu n t Stnr Ts Grp St. Stnr Worlo Formt. Son, sn rl tr ontns tns o tousns o os, ltrn out os n sltn t rst-n os orn to vrous rtr s ssntl or sltn usl worlo; GRENCHMARK n us to sp n t rl worlo tr. Tr, rl trs nnot usully rply on notr systm tn t on on w ty wr qur, n vn on t sm systm t s n. T lttr ppns mor otn n t s o rs, w r nturlly volvn wt tr mlwr, n tror r vry ynm. GRENCHMARK n sl vrous spts o t worlo,.., t rqust rsours, or t o runtms, n n lp llvt ts prolm [7]. 3.3 Moln worlos wt GRENCHMARK GRENCHMARK ors support or t ollown worlo moln spts. Frst, t supports untry n ompost ppltons, n snl-st n o-llot os. Son, t llows t usr to n vrous o ntr-rrvl tms s on wll-nown sttstl strutons. Bss t Posson struton, us trtonlly n quu-s systms smulton, GRENCHMARK lso supports unorm, norml, xponntl n ypr-xponntl, Wull, lo norml, n mm strutons. Tr, t llows t worlo snr to omn svrl worlos nto snl on (mxn). Ts llows or nstn t nluson o ursts, y omnn sort worlo wt mny os pr tm unt wt lonr on, omprsn wr os pr tm unt. An tonl us o worlo mxn s n wt- nlyss tt vluts wt wll ppn to r ommunty ts rsours woul sr wt notr roup o usrs. In ts s, t worlo molr n mx t typl worlo o t two ommunts n vlut wtr t systm n support ot, unr vrous o ptn n xuton pols. 3.4 T GRENCHMARK pross W n two us ss or t GRENCHMARK rmwor: n rl worl n n smultons. A worlo nr- 1 Post-prouton Anlyz rsults Inr mtrs Rport prormn Worlo srpton Applton typ 1 Syntt Gnrt Worlo 5 r srpton 2 Worlo Dt Applton typ n Is strr, stout St output JoSumt stts strr, stout Worlo Output 4 Jo 1 JoSumt runnr RM St 1 Sumt Worlo Jo 2 JoSumt onor-o-sumt RM St n St 2 RM Gr Rsour Mnr SGE,PBS,... Jos rt Jo n Fur 2. T GRENCHMARK pross. # Fl-typ: txt/wl-sp #Jos Typ StTyp Totl StIno ArrvlTmDstr OtrIno 25 ssr snl 1 *:? Posson(12s) StrtAt=s 25 ssro snl 1 *:? Posson(12s) StrtAt=6s 25 smp1 snl 1 *:? Posson(12s) StrtAt=3s,ExtrnlFl=smp1.xn 25 smp1 snl 1 *:? Posson(12s) StrtAt=9s,ExtrnlFl=smp2.xn Fur 3. A GRENCHMARK worlo srpton xmpl. t y GRENCHMARK usn only syntt n wll stu ppltons n qully us n ot ss. Fur 2 pts t typl pross o usn t GRENCHMARK rmwor n rl nvronmnt. Frst, t usr srs t worlo to nrt, s ormtt txt l (1). Bs on t usr srpton, on t nown pplton typs, n on normton out t r sts, worlo s tn nrt y GRENCHMARK (2), n sumtt to t r (3). T r nvronmnt s rsponsl or xutn t os n rturnn tr rsults (4). T rsults nlu not only o outoms, ut lso tl sumsson rports. Fnlly, t usr prosss ll rsults n post-prouton stp (5). T us o GRENCHMARK or smultons s smlr, wt stps (3) n (4) possly omn. T most ult stp n usn GRENCHMARK s stp (1): srn t worlo. To s ts ts, w v sn smpl n xtnsl lnu; t worlo snr s only onrn wt t ulty o sn- Gr Prons o t Sxt IEEE Intrntonl Symposum on Clustr Computn n t Gr (CCGRID'6)

5 StTypsWtWts=nonx/3;x/5 StsWtWts=s1;s2;s3;s4 NComponntsWtWts=1/5.;3/16.2;5/5.;1;15;2/1.;4/1.;8 TotlCPUsWtWts=2/2.;4/3.;5;8;1;16;2;32... Fur 4. A GRENCHMARK worlo spton lnu xtnson. n rprsnttv worlos, rtr tn ow to sr tm. Fur 3 sows t worlo srpton or nrtn t mr+ tst, omprsn 1 os o our rnt typs. T rst two lns r ommnts. T nxt two lns r us to nrt squntl os o typs ssr n ssro, wt ult prmtrs. T nl two lns r us to nrt MPI os o typ smp1, wt prmtrs sp n xtrnl ls smp1.xn n smp2.xn. All our o typs ssum n rrvl pross wt Posson struton, wt n vr rt o 1 o vry 12 sons. T rst o o typ strts t tm sp n t worlo srpton wt t lp o t StrtAt t. For MPI os, t sp xtrnl l (t ExtrnlFl) ontns otr pplton-sp prmtrs (s lso Ston 3.5 n Fur 4). 3.5 Extnn t GRENCHMARK rmwor GRENCHMARK s n sn wt n nrmntl ppro n mn, n ltts utur xtnsons. T rmwor sn n sly xtn, or nstn y n vrous worlo nrton notons (.., usrs, vrtul ornztons). GRENCHMARK lso ors smpl plu-n systm, w n us to unt nrtors (nw pplton typs) n prntrs (support or otr r rsour mnrs). To xtn t worlo nrton pross, t usr s rst to xtn worlo spton lnu, n tn to wrt plu-n tt uss t lnu xtnson; t only rqurmnt s tt t xtnson lnu s s on Ky=Vlu sttmnts. A l wrttn n n xtnson lnu s utomtlly prs, n t t s vll to t usr wn t plu-n s nvo. For t lnu xtnson, GRENCHMARK ors mnsms to prs smpl vlus (.., oolns, strns, ntrs, n lots), n mor omplt onstruts (.., lsts, n lsts wt wts); t usr n lso to prs t xtnson lnu sttmnts rtly. Fur 4 sows n xmpl o l wrttn n n xtn lnu, or nrtn worlo wt ollot os. Ln 1 ns t typ o o-llot os to nrt s lst wt wts; nonx os (multst os wt unsp xuton sts) v lowr wt tn x os (mult-st os wt sp xuton sts). Ln 2 ns t possl xuton sts, s lst wt wts. A ult wt o 1. s utomtlly ssn to t lmnts or w t usr s not sp t wt. Lns 3 n 4 us lsts wt wts to n t numr o omponnts n t numr o prossors or nrt os. Jos wt 1 omponnt n 4 prossors r prrr. 4 T rsults Ts ston prsnts t rsults otn wt GRENCH- MARK. T xprmnts r not u to sow ull nlyss o rtn tur; nst, w try to sow ow GRENCHMARK n us or vrous ols. T mor rn twn t two ppros s n t wy rsults r nlyz, u to t rn n ols;.., n Tl 2 w sow summry o runtms or tr ppltons (prormn tstn), ut not t tl nlyss rqur or t omplt prormn rtrzton o ts ppltons (prormn nlyss). W n sussul o to o tt qurs ts rqust rsours, runs, nss, n rturns ll rsults wtn t tm llow or t worlo. Unlss otrws stt, w us t suss rt o t os s t prormn mtr or our xprmnts. T rson s twool: w n tt or urrnt rs t lty to rlly run os s vn mor mportnt tn rw prormn, n w ru (s on t rsults sown n Tl 2) tt GRENCHMARK rsults n lso us to xtrt otr mtrs (rom t mny vll, s or nstn [8]). 4.1 Exprmntl stup W us t DAS systm 5 s n xprmntl nvronmnt. T DAS systm omprss 5 lustrs, wt 32 up to 72 ul-prossor SMP nos. A sr l systm (NFS) s us wtn lustr. E lustr s on ntry pont (twy), on w usrs n lo n sumt os, or rtrv tr os output. T DAS systm os not srmnt twn lol n rmot usrs, ut s rnt rsour lloton pols pnn on t os rqust runtms. GRENCHMARK ws us to nrt n sumt t worlos. W us ot mol n rl (tr-s) worlos to nrt t tst worlos. E worlo ws sumtt n t norml DAS worn nvronmnt, tus n nlun y t roun lo nrt y otr DAS usrs. Som os oul not ns n t tm or w ty rqust rsours, n wr stopp utomtlly y t KOALA sulr. Ts stuton orrspons to usrs unr-stmtn ppltons runtms. E worlo rn twn t sumsson strt tm n 2 mnuts tr t sumsson o t lst o. Tus, som os not run, s not nou r rsours wr vll urn t tm twn tr sumsson n t n o t worlo run. Ts stuton s typl or rl worn nvronmnts, n n l to run n stop t wor- 5 Dstrut ASCI Supromputr, ttp:// Prons o t Sxt IEEE Intrntonl Symposum on Clustr Computn n t Gr (CCGRID'6)

6 Tl 1. T prormn tstn rsults. Typs o #o Componnt Suss Worlo Appltons CPUs # sz Rt mr1 syntt, sq % smp1 syntt, MPI % s1 N Quns, Is % lo orn to t usr sptons sows som o t plts o GRENCHMARK. 4.2 Gr systm nlyss Ts ston sows ow GRENCHMARK n us or two typs o systm nlyss: prormn tstn (n nlyss), n wt- nlyss Prormn tstn W onsr prormn tstn s n w t usr wnts to tst t prormn o on or mor o s ppltons. Ts stuton ours wn vlopn n pplton, wn tstn on or mor systm sn optons. In ll o ts stutons, t usr wnts to nrt t worlo wt s lttl nput s possl, sumt t nrt worlos, n v tl rport on t otn rsults wtn rtn tm-rm. Tl 1 sows t strutur o t tr nrt worlos, n t tl suss rt or runnn tm. W us tr typs o ppltons: ssr (syntt squntl), smp1 (syntt prlll, MPI), n n N Quns solvr (syntt prlll, Is). E nrt worlo ontns on pplton typ, n omprss 1 rnt nstns. For syntt ppltons, w us omntons o prmtrs tt woul p t run-tm o t ppltons unr 3 mnuts, unr optml ontons. For t Is os (worlo s1), 1-2% o t ppltons v svrly unrstmt runtm rqusts; ts orrspons to t stuton wr som usrs svrly unrstmt t runtms o omplx ppltons [14]. E o rqusts rsours or tm low 15 mnuts. T omnton o run-tm n rsour rqust sttns nsurs tt som ppltons woul l u to t usr s own ult. T totl sumsson tm o ny worlo s pt unr two ours, to sow ow GRENCHMARK n us n tm-onstrn tst stuton. To stsy typl r stutons, os rqust rsours or 1 to 15 omponnts. As t DAS s only 5 sts, os wt mor tn 5 omponnts wll v svrl omponnts runnn t t sm st. For prlll os, tr s prrn or 2 n 4 omponnts. For mult-omponnt os tr s prrn or 2, 4, n 16 prossors. Vrous ntr-rrvl tm strutons r us to nrt t sumsson tm o worlo os. Componnt rqusts r tr x (spyn t nm o r st) or nonx (lvn t sulr to ). Tl 2. A summry o tm n run/suss prnts or rnt o typs. Jo Turnroun [s] Runtm [s] Run+ nm Av Rn Av Rn Run Su ssr % 97% smp % 81% NQuns % 85% Tl 3. T rsults or t rst s o wt nlyss. Sumt Jo sz rt vs. #o (# o Sumt Worlo ornl os CPUs) Errors DAS2-FS3-1x 1x % DAS2-FS3-1x 1x % DAS2-FS3-25x 25x % DAS2-FS3-5x 5x % DAS2-FS3-1x 1x % T lowr prormn o prlll os (worlos smp1 n s1) wn ompr to on-prossor os (worlo mr1), s us y t prlll os n to llot rsour sts, s oppos to llotn snl rsours. Is os lso sur rom t smll tm lmt oupl wt nurt stmtons. T turnroun tm o n pplton n vry rtly (s Tl 2), u to rnt prmtr sttns, or to vryn systm lo. T vrtons n t pplton runtms r u to rnt prmtr sttns. As xpt, t prnt o t ppltons tt r tully run (Tl 2, olumn Run) pns vly on t o sz n systm lo. T suss rt o os tt run sows lttl vrton (Tl 2, olumn Run+Suss) Wt- nlyss W onsr tr us ss or llustrtn GRENCH- MARK s plts or wt- nlyss : systm n, r ntr-oprlty, n spl stutons. In t rst two ss w ssum tt t nvronmnts unr srutny v n torouly stu n tr worlos mol. Frst, w onsr t s wr tstn nvronmnt woul pl unr (mu) mor strn. Our tstn nvronmnt, t DAS systm, s tully ot-swpp ts rsour mnr, us t oul not op wt t nrsn numr o o sumssons. T wor o ll t DAS ommunty ws ntvly t or pro o two ws. Su n oul v n prvnt t quston Wt t urrnt usrs woul sumt 1 tms mor os n t sm mount o tm? Or 5 tms, or 1 tms... woul v n nswr t t systm nstllton, or urn qut pro. Furtrmor, t prot sponsorn ts rsr (s Anowlmnts) s s ts Prons o t Sxt IEEE Intrntonl Symposum on Clustr Computn n t Gr (CCGRID'6)

7 Tl 4. T rsults or t son s o wt nlyss. #o Componnt Suss Worlo os no. sz Rt DAS2-FS3 + OSC % DAS2-FS3 + CTC % DAS2-FS3 + SDSC % Systm utlzton [%] Lo on t DAS2 - s4 Totl utlston % G RENCHMARK % Av. Utlston % mn otv mn t DAS nrstrutur sly vll to t Dut ms, n rqurs ts wt- quston to nswr or t nw sttns r rls to t pul. W t torouly stu tr [1] o t DAS systm n r-run t n t nw nvronmnt. T tr ws ror wn t prvous rsour mnr ws stll n pl, n ontns ovr 425, os run trouout 23. To lmt t tstn pro, w onsr only t os sumtt to on o t v DAS lustrs (ovr 65,), lmt t worlos to t most 1 os, n nsur tt t rsultn worlos v sumsson spn o mxmum tr ours. All ts rqurmnts wr mt trou t worlo nton lnu; no tool xtrnl to GRENCH- MARK,.., MySQL, ws us. W lso sl t os sumsson tms to m tm 1, 25, 5, n 1 tms smllr tn t ornl sumsson tms. Tl 3 tls t ltr n sl trs n sows t prnt o t sumt rrors, rom t totl numr o sumtt os. W onlu tt t nw rsour mnmnt systm (nlun KOALA) n nl n up to 1 tms nrs n t sumsson rt, or t usr s rtrz y t nput trs. Son, w onsr t s wr our tstn nvronmnt woul lso us y t usrs o notr nvronmnt n w wnt to n out wt s t suss rt o t os sumtt y ts omn ommunts?. Ts stuton ours wn rom two xstn prouton nvronmnts on nvronmnt s put tmporrly or prmnntly out o prouton, n only on nvronmnt rmns to run t sumssons o os rom ot usr ommunts. For t xtn us s n w two nvronmnts sr tr rsours, sus or t usr to run t nrt worlo on ot systms t t sm tm; or ts purpos, GRENCHMARK llows nrt worlo to urtly rply on rnt systms (vn tos systms r smult). W too t sm tr s n t rst wt- nlyss s, n omn t wt tr o t OSC, t CTC, or t SDSC 96 trs rom t Prlll Worlos Arv (PWA). W slt ust t os wt runtms low 9 sons. W n tr o st s st onsutv os rom tr or w t sumsson ntrvl s 3 ours. To solv t prolm o not sltn t most mnn, nor t lst mnn tr o st, w rst slt t top 1 sts, sort y tr numr o os (sz). Tn, w omput t vr numr o os or t top 2 Sp 1 Sp Tm [s] Fur 5. T utlzton o t DAS s4 lustr or ursty sumssons. 1 sts, n w oos t st wos sz s t losst to ts vr. Tl 4 sows t tl strutur o t nrt worlos, n t suss rt or runnn tm. T osrv suss rt ws ov 7% or ll tsts. W onlu tt t nw DAS (nlun KOALA) n nl t propos omntons o ommunts, wt rsonl suss rt or sumtt os. Tr, w onsr t s wr t usr wnts to tst t outom o t systm n sunly sut to lr numr o sumtt os (ursts). Usn worlos s on syntt ppltons, w nrt two ursts, n or roun lo, lln 25% n 5% o t systm s pty, rsptvly. Durn t tsts, t roun lo nrt y otr usrs ws twn % n 5%. W rstrt our tsts to on o t DAS lustrs (s4). Fur 5 sows t systm utlzton rp urn t pro wn t two urst worlos r sumtt. T systm prorm orrtly unr strn; t mxmum systm utlzton s 95%, o w 9% s nrt y t urst worlos. Two sps n lrly nt on t utlzton rp. As xpt, runnn t rst urst worlo (sp 1) ts lonr, u to t r roun utlzton o t systm. 4.3 Funtonlty tstn n r nvronmnts Ts ston prsnts two ss or sown ow GRENCHMARK n us or untonlty tstn: systm untonlty tstn, n pro systm tstn Systm untonlty tstn W onsr t s wr t usr wnts to tst t systm s plty to rlly run vrous typs o os t t sm tm. W omn t worlos us n t s o prormn tstn n otn mx worlos. Prons o t Sxt IEEE Intrntonl Symposum on Clustr Computn n t Gr (CCGRID'6)

8 Tl 5. T rsults or systm untonlty tstn. Typs o #o Componnt Su Worlo ppltons CPUs no. sz Rt mr+ syntt, sq.&mpi % s+ vrous, Is % untry mr+ & s % Tl 6. Rsults o t omprson o suss rts or snl-st vs. o-llot os. #o Componnt Suss Worlo CPUs no. sz Rt snl-st % ollot % ollot % Tl 5 splys t omposton o t two xprmntl worlos, n t suss rts or tr xuton. T mr+ worlo omprss mx o prlll n squntl syntt ppltons lso us n worlos mr1 n smp1 (s Tl 1). As xpt, t suss rt o mr+ s lowr tn or mr1, ut r tn or smp1. T suss rt lututs wt t numr o smp1 os: t lowr ts numr, t lowr t lur rt (squntl os r mor stl). T sm onsrtons n ppl or t s+ n untry worlos. For t s+ worlo, t suss rt s 15% lowr tn norml u to svr runtm unrstmtons (s lso Ston 4.2.1). W onlu tt t DAS n KOALA n rlly run t GRENCHMARK os Pro systm tstn W onsr s wr t usr wnts to prolly tst systm. Ts typ o tstn lps ntyn systm prolms, or n us or otnn t urrnt prormn o vrous systm omponnts. W sul st o smpl tst worlos or ly sumsson, rom vrous DAS twys. Ts lp to quly nty n solv rtl ploymnt prolm: t onurton l o KOALA on on o t twys ws norrt n us t sumsson tst to l; lt unorrt, usrs sumttn os rom tt twy woul v n unl to sumt som o tr os trou tt twy. 4.4 Comprn r sttns Ts ston prsnts two stutons n w GRENCH- MARK n us or omprn r sttns Snl st vs. o-llot os W onsr omprson twn t suss rts o snl-st n o-llot os n r nvronmnt Tl 7. Comprson o suss rts or untry vs. ompost os wt or wtout xuton ult tolrn. # o Jos/ Suss rt wn ult tolrn s Worlo Su-Jos OFF ON untry 1/1 9% 1% ompost 1/18 7-9% 1% wtout rsrvton plts. W vs tr worlos, on wt snl-st os, on wt t sm os o-llot t vrous sts, n on wt lrr os o-llot t vrous sts. Tl 6 sows t tl strutur n t suss rt or t tr worlos. T worlos omprs only syntt MPI ppltons. T snl-st os (worlo snl-st) v r suss rt tn o-llot os (worlos ollot n ollot+), n ts s y ovr 2%. Ts s u to t tom rsrvton prolms o o-lloton: lol usrs n qur rsours slt or o-lloton ust or ty r lm y t o-llotn sulr. Sn t vr lo n t DAS systm s low (.., low 25% [1]), smll n lr ollot os v lmost t sm suss rt (nou mns r vll) Untry vs. ompost os W onsr omprson twn t suss rt o untry n ompost ppltons n r nvronmnt wtout rsrvton plts, n wt or wtout ult tolrn. W onsr only DAG-s ompost ppltons or w t su-os r untry ppltons. W n o tt n run s o or w ll ts pnns v n mt. Sn KOALA os not support t xuton o ompost os, w ult smpl xuton tool, w xuts os tt n run s soon s possl. T xuton tool lso llows or lxl ult tolrn sms; w us r smpl rtry l mnsm wt su-o rnulrty. For rnt worlow typs, xuton tools, n mnsms, w rr to [15]. Tl 7 sows t suss rt or untry n ompost os, wt or wtout ult tolrn. Wtout ult tolrn, t suss rt o ompost os rops rmtlly t rst su-os n t DAG s topolol sort l. Wt ult tolrn, or systm wt suss rt or os,.., t DAS, t smpl rtry mnsm yls omplt rllty n runnn orrt os. 5 Rlt wor A snnt numr o prots v tr to tl t Gr prormn ssssmnt prolm rom rnt nls: moln worlos n smultn tr run unr vrous nvronmnt ssumptons [14, 6, 4], ttmptn to Prons o t Sxt IEEE Intrntonl Symposum on Clustr Computn n t Gr (CCGRID'6)

9 prou rprsnttv st o r ppltons l t NAS Gr Bnmrs [9], rtn syntt ppltons tt n ssss t sttus o r srvs l t GRASP prot [5], n rtn tools or lunn nmrs n rportn rsults l t GrBn prot [12]. Compr to ts ppros, GRENCHMARK s rnt sn prnpls: () t os not ttmpt to mpos r nmr, only t mns trou w su nmr my ult; () t trs to or lr st o ppltons, vn worlo moln turs, n t lty to rply xstn worlo trs; n () t s n splly ult so tt t n us or mor spts tn ust r prormn vluton (s Ston 4). From t non-r worl, GRENCHMARK rsmls t Ap JMtr [1], worlo nrtor, sumttr, n nlyzr or w n ts ppltons. 6 Conluson n utur wor Ts ppr s prsnt GRENCHMARK, rmwor or syntt r worlos nrton n sumsson. T rmwor supports vrous worlo moln prmtvs, oms wt ovr 35 syntt n rl ppltons, n s portl, lxl, n xtnsl. W v sn n mplmnt GRENCHMARK, n v ploy t n t DAS, our 4-prossors r nvronmnt. W v sown vn tt GRENCHMARK n sussully us or r systm nlyss, untonlty tstn n r nvronmnts, n omprn rnt r sttns. W v prsnt vrous xmpls o ow GRENCHMARK n us or prormn tsts, wt nlyss, systm untonlty tsts, pro systm tsts, snl vs. o-llot os omprson, n untry vs. ompost os omprson. W v sown ow ts tsts n xtn or rnt ols tn tos o ts ppr,.., otr prormn mtrs n otr typs o tsts. Lst, ut rtnly not lst, GRENCHMARK ws nstrumntl n mn t KOALA r sulr rll on t DAS, n rlsn t or t DAS usr ommunty. W r urrntly xtnn GRENCHMARK wt support or mlll os. For t utur, w pln to us GRENCH- MARK or tstn n omprn mor r sttns, ot n smultons n rl stutons. Avllty T ol GRENCHMARK w st, nlun oumntton n rly vll struton, s lot t: ttp://rnmr.st.w.tult.nl/. Anowlmnts Ts wor ws rr out n t ontxt o t Vrtul Lortory or -Sn prot ( w s support y BSIK rnt rom t Dut Mnstry o Euton, Cultur n Sn (OC&W), n w s prt o t ICT nnovton prorm o t Dut Mnstry o Eonom Ars (EZ). Rrns [1] Ap Dvlopmnt Tm. Ap JMtr. URL ttp://rt.p.or/mtr/, 26. [2] H. E. Bl t l. T strut ASCI supromputr prot. Oprtn Systms Rvw, 34(4):76 96, Otor 2. [3] F. Brmn, A. Hy, n G. Fox. Gr Computn: Mn T Glol Inrstrutur Rlty. Wly Pulsn Hous, 23. [4] A. I. D. Buur n D. H. J. Epm. Tr-s smultons o prossor o-lloton pols n multlustrs. In Pro. o t 12t IEEE HPDC, ps 7 79, 23. [5] G. Cun, H. Dl, H. Csnov, n A. Snvly. Bnmr pros or r ssssmnt. In IPDPS, 24. [6] C. Ernmnn, V. Hmsr, U. Swlson, R. Yypour, n A. Strt. On vnts o r omputn or prlll o suln. In CCGRID, ps 39 49, 22. [7] C. Ernmnn, B. Son, n R. Yypour. Sln o worlo trs. In D. G. Ftlson, L. Ruolp, n U. Swlson, tors, JSSPP, volum 2862 o LNCS, ps Sprnr, 23. [8] D. G. Ftlson n L. Ruolp. Mtrs n nmrn or prlll o suln. In D. G. Ftlson n L. Ruolp, tors, JSSPP, volum 1459 o LNCS, ps Sprnr, [9] M. Frumn n R. F. V. r Wnrt. Ns r nmrs: A tool or r sp xplorton. Clustr Computn, 5(3): , 22. [1] H. L, D. Grop, n L. Woltrs. Worlo rtrsts o mult-lustr supromputr. In D. G. Ftlson, L. Ruolp, n U. Swlson, tors, JSSPP, LNCS, vol.3277, ps Sprnr, 24. [11] H. Mom n D. Epm. Exprns wt t ol o-llotn sulr n multlustrs. In Pro. o t 5t IEEE/ACM Int l Symp. on Clustr Computn n t GRID (CCGr25), My 25. [12] G. Tsouloups n M. D. Dos. GrBn: A worn or r nmrn. In P. M. A. Sloot, A. G. Hostr, T. Prol, A. Rnl, n M. Bu, tors, EGC, volum 347 o LNCS, ps Sprnr, 25. [13] R. V. vn Nuwpoort, J. Mssn, G. Wrzsns, R. Homn, C. Jos, T. Klmnn, n H. E. Bl. Is: lxl n nt v-s r prormmn nvronmnt. Conurrny & Computton: Prt & Exprn., 17(7-8): , Jun-July 25. [14] A. M. Wl n D. G. Ftlson. Utlzton, prtlty, worlos, n usr runtm stmts n suln t IBM SP2 wt lln. IEEE Trns. Prlll Dstr. Syst., 12(6): , 21. [15] J. Yu n R. Buyy. A txonomy o snt worlow systms or r omputn. ACM SIGMOD R., 34(3):44 49, 25. Prons o t Sxt IEEE Intrntonl Symposum on Clustr Computn n t Gr (CCGRID'6)