A Scalable Double In-memory Checkpoint and Restart Scheme towards Exascale

Transcription

1 A Sll Doul In-mmory Ckpont n Rstrt Sm towrs Exsl Gnn Zn, Xn N, Lxmknt V. Klé Dprtmnt o Computr Sn Unvrsty o Illnos t Urn-Cmpn Urn, IL 6181, USA E-ml: {zn, xnn2, kl}@llnos.u Astrt As t sz o supromputrs nrss, t prolty o systm lur rows sustntlly, posn n nrsnly snnt lln or sllty. It s mportnt to prov rsln or lon runnn ppltons. Ckpont-s ult tolrn mtos r tv ppros t ln wt ults. Wt ts mtos, t stt o t ntr prlll pplton s kpont to rll stor. Wn lur ours, t pplton s rstrt rom rnt kpont. In prvous work, w v monstrt n nt oul n-mmory kpont n rstrt ult tolrn sm, w lvrs Crm++ s prlll ots or kpontn. In ts ppr, w urtr optmz t sm y lmntn svrl ottlnks us y srlz ommunton. W xtn t n-mmory kpontn sm to work on MPI ommunton lyr, n monstrt t prormn on vry lr sl supromputrs. For xmpl, wn runnn mllon tom molulr ynms smulton on up to 64K ors o BluGn/P mn, t kpont tm ws n mllsons. T rstrt tms wr msur to lss tn.15 sons on 64K ors. I. INTRODUCTION On onrn o prormn omputn or xsl s t lty to tolrt ults. Evn tou toy s supromputrs r ompos o rll prts, t mn tm twn lur (MTBF) stll rops own s t numr o prossors nrss. Jur, t 3r rnk supromputr on top 5 toy, 2.33 vr lurs pr y urn t pro rom Auust 28 to Frury 21. For toy s snt smulton runnn or mny ys, su lurs r tstrop. On ommon soluton to support ult tolrn r t sk-s kpont/rstrt sms. In ts mtos, ppltons or HPC systm kpont t stt o t ntr prlll pplton n or rstrt to rll mum, typlly NFS ult r sp. Howvr, prol kpontn to su slow stor n xpnsv, or xmpl, t oul onsum up to 2% o t pplton tm [1]. Tr r two mn mtos to strt kpontn: systm or pplton ntt kpont. Systm s kpont my tk 2 to 4 mnuts to kpontn or t st mns on t top 5 lst (28) [2], [3]. Tror, systm lvl kpontn to t l systm s lrly mprtl t xsl. Applton-lvl kpontn n lp run kpont sz, ut s stll llnn n trms o t sllty t xsl, n pls tonl urn on pplton prormmr. A runtm-lvl kpontn sm rus t urn on t prormmr, splly y utomtn t protool or trrn kponts, n rryn out rovry. In our prvous work [4], [5], w xplor n n-mmory kpont/rstrt sm n ult tolrnt CHARM++ [6] n Aptv MPI [7] runtm systms. T protool os not rly on ny rll stor or kponts. T rstrt protool llows pplton to ontnu to run on t survvn prossors tr rs wtout ull stop. T protool uss lol mmory or sk or kponts, n n lvr t sp ommunton ntwork to sp up t kpontn pross. T s s to prolly rt two kponts or ps o pplton t npsult n CHARM++ ots. On kpont s stor n lol stor (mmory or lol sk), n t otr s stor on rnt no, ll uy no. On lur, t survvn nos rstors tr ot t rom t lol kponts, wrs t ots on osn spr no (to rpl t l on) r rstor usn kponts stor t uy no. Ts mto s pl o tolrtn ll snl lurs, n most multpl lurs, l nos r not uy to otr. Altou ts sm os not prov n nlll mto o ult tolrn, t soul sunt to pply to vry lr mns s snl lurs r t most ommon lur n toy s HPC systm. For xmpl, 95% o t lurs on TSUBAME r snl lur. T two kponts onsttut mmory ovr or ts sm, ut (somwt surprsnly) ts s tolrl or lr lss o ppltons: tos tt v smllr mmory ootprnt t kpont. Ts nlu molulr ynms, N- oy os, rtn quntum mstry (nnomtrls os), t. For otrs, t sm rls on utur lol stor (or o ours, lol l systm s ll k). In ts ppr, w stuy t tnqus rqur n t oul n-mmory kpont n rstrt sm or vry lr sl. W xmn t ovrs o t mplmntton tt oul potntlly lmt sllty, n urtr mprov t mplmntton y optmzn ommunton (splly t olltv ommunton). On ostl or monstrtn ult tolrn on MPI ppltons s tt t quun systm on supromputrs klls t ntr o wn pross ls. To work roun ts

2 lmtton, w vlop ult nton sm tt mms lur o pross wtout tully klln t. Ts llows us to monstrt t ult tolrn sm wt MPI on vry lr sl supromputrs. T rst o ts ppr s ornz s ollows. Ston II srs t oul n-mmory kpont/rstrt protool s kroun. T tnqu w us to monstrt t or MPI ppltons n urtr optmzton o t kpont/rstrt sm to xsl s prsnt n Ston III. Prormn rsults o t optmz ult tolrn sm on up to 64K ors r prov n Ston IV. W suss som rlt work n Ston V, n nlly, Ston VI onlus t ppr. II. BACKGROUND In ts ston, w summrz sn o sll nmmory kpont-s ult tolrn sm trtn vry lr sl prlll ppltons. T supportn prlll runtm systms r Crm++, mss rvn runtm systm, n Aptv MPI [7], n mplmntton o MPI on top o Crm++. Ts ult tolrnt runtms tk vnt o t mrtlty ots n trs. A. Runtm Support or Ckpont/Rstrt T ult tolrnt runtm systm supports kpontn o pplton t n two lvls: ully utomt kpontn or lxl usr-ontroll kpontn y tonl lpr untons. Aptv MPI [8] runs MPI prosss n lt-wt trs, w r sr or kpont n rstrt to nl ompr to prosss. Tr mrton urn rstrt woul rs t prolm o pontr rrn. Isomllo [8], [9] s us to solv ts prolm or ully utomt kpontn, smlr to t tnqu n t P M 2 systm [1]. Isomllo rsrvs rn o vrtul rss sp or ll t prossors. Durn kpontn, vrtul rsss o t MPI trs or ots n t t ssot wt tm r sv utomtlly. A ot or tr n tn rstor on ny prossor sn t llot t n rstor wtout nn ts rss. Anotr opton s tt usrs n wrt tr own lpr untons to pk n unpk p t or kpontn n rstorn n ot. Ts s somtms usul n run t sz o t nvolv n kpontn n rstorn. Applton vloprs oul us pplton sp knowl to pk only t lv vrls t t tm o kpontn, or us omplr to utomt ts [11]. Ts mto rus t t mount to kpont, n so kpontn oms str. B. Bs Doul In-Mmory Ckpont/Rstrt Sm Ckpont/rstrt sm rqurs nos to rquntly sv tr omplt stt to stl stor or t mmory o notr no. T optmum tm twn kponts s n nlyz lswr [12], [13]. T n-mmory kpontn sm [4] ntrou t o sklss kpontn tt kponts t n mmory. It uss oornt kpont strty, w rqurs PE PE1 PE2 PE3 F. 1: In-Mmory Snl Ckpont PE PE PE1 PE2 PE3 PE2 rs, lost on prossor PE1 PE3 ot kpont 1 kpont 2 rstor ot F. 2: In-Mmory Snl Ckpont ppltons to v synronzton pont wr ty oul strt lol olltv oprton to kpont. In orr to nl on lur t tm, ommon s snro, on kpont o t pplton stt n t mmory o rnt prossor s not sunt s llustrt n Fur 1. In ts snro wt 4 prossors, Crm++ ot (rprsnt s rl) kponts only on opy o ts kpont (rprsnt s trnl). Wn prossor 2 rss, t kponts or ot n n mmory o tt prossor r prmnntly lost, so w ouln t rovr rom t kpont. Ts susts tt t lst two ops o t kpont t rnt lotons r n. In prtulr, w opt n n-mmory oul-kpontn sm w n tolrt t lst on lur t tm. Fur 2 llustrts n xmpl o ts sm. T top l o t ur sows t snro or on prossor rss. E rl rprsnts n ot n kpont, wl trnl n squr rprsnts ts rst n son kponts. W ll ts two prossors uy prossors or t kpontn ot. Not tt on o t two uy prossors n t sm prossor wr t ot rss. Ts n lp ru t kpontn ovr, sn t

3 kpontn s slly lol mmory opy, w s mu str tn ssn mmory o rmot prossors. Ovrll, ompr to t trtonl on-sk kpontn, n-mmory kpontn sm uss mmory s stor n strut wy, tkn vnt o t sp ntronnt, w tns to mor nt. T rstrt prour s ntt y rs o pysl prossor. On lustrs, t rs ttor n t runtm systm tts t rs trou rokn pp sokt rrors. Wn t rstrt prour s ntt, ll survvn prossors xmn t kponts n tr mmory n k or mssn uy prossors. A nw prossor s osn (w n tr spr prossor, or runnn prossor) to rpl t rs prossor n t ltst kpont t s op to tt prossor to mntn t oul kponts. On o t two uy prossors s tn rsponsl or rstorn t orrsponn ots rom ts kponts n mmory. At rstrt, t rplmnt prossor s rom runnn prossor, tn lo mln my our sn tt prossor rstors mor kponts. Ts n x y lo lnn ps tr rstrt [4]. T ottom l o t Fur 2 llustrts snpsot o t ots n tr kponts strut on prossors tr rovry s omplt. T lost kponts on t rs prossor 2 r rovr to prossor 3 n prossor rsptvly. Prossor 3 s osn to rstor prossor 2 s ots(,) lolly to vo ommunton ovr, sn Prossor 3 s prossor 2 s ornl uy prossor. Our protool nsurs t rovry rom snl no lur n w n rovr rom multpl onurrnt lurs t rs prossors r not us to otr. III. OPTIMIZATIONS FOR SCALABILITY In prvous work [4], w v sown tt t s sm v oo prormn on lustrs o mor tn unr ors [4]. Howvr, wn w tst ts sm on Blu Gn/P mn wt tns o tousns o ors, w oun ot t kpontn n rstrtn tm nrss lmost lnrly s t numr o prossors nrss, s sown n Fur 3 n 6, w s not sll. Altou t solut prormn s stll surprsnly oo (or xmpl, rstrt on 64K prossors only took lss tn 4 sons), or xsl systms, t mt stll prolm wn tr r, or xmpl, mllons o ors. Ts motvts t work n ts ppr to urtr optmz t oul n-mmory kpont/rstrt sm or xsl mns. A. Optmzton Tnqus Stl mss nln: Sn t oul n-mmory sm rstrts pplton on t ly wtout tully rstrtn t prlll o, t t nnn o t rstrt ps, wn t pplton s rstor to rnt kponts, ol msss tt r snt or kponts r rstor r stll n trnsmsson or ur, n possly mx wt systm msss u to t onon prlll rstrt. Evn mor omplt, t s possl or t stl msss to vntully lvr on no tr kponts r rstor. Tror, t s rtl to sr ll stl msss rom t rs. To rntt t stl msss wt t ll msss snt tr rs, n po numr s us n our oul n-mmory kpont sm. Howvr, ult tton n ult nnounmnt to ll nos ours n prlll wt t ontnuous xuton o t survvn nos. Wn omputton only pns on norn ors, t omputton on tos ors my not t n kp on vn tr no s rs untl t ult notton mss nlly rrvs. Durn ts pro o tm, mor stl msss my nrt rom tos ors, w nrss t urn o srn ts stl msss. On vry lr sl mns, ln wt stl msss systm w my xpnsv, tror, t s rtl to trottl t xuton o t prorm s soon s possl tr rs s tt to prvnt propton o t stl msss. In t optmz sm, w ntrou nw ps n rstrt w s t to trottln t stl msss. Ts ps strts mmtly tr ult s tt. All nos ntr stt n w ty sr ll rv msss untl qusn s tt, w mns tr r no stl msss. T qusn tton s n nt tr-s lortm tt s t omplxty o O(loP ), wr P s t numr o nos. Optmzn smll msss usn strmn tnqus: Durn rstrt, mny smll ookkpn msss r snt to upt t runtm systm out t mrtl ots. For xmpl, or ot rstor, smll mss ontnn ts nw loton s snt to ts om prossor. On vry lr systms, ts xtrmly lr numr o smll msss om prolm - t uss tr m n rtly slows own t rstrt pross. In t optmz vrson, w ppl strmn optmzton tnqu tt omns smll msss snt to t sm stnton prossors nto on r mss. Optmzn olltv ommunton: In t ornl sm, svrl rrrs r n urn kpont n rstrt to nsur t orr o kpont/rstrt pss, or xmpl, t rovry o t rry lmnts rom kponts ppns only tr rmovn ll t stl ot t n mmory o ll prossors. Ts rrrs rt sllty lln. Altou, nt spnnn tr-s ruton mplmntton xst n CHARM++, t n not sly us n ts s. Ts s us t mplmntton n not nl t s wn tr s rs prossor n t tr n t nonsstnt ruton squn numr us y t lur. Tror, t ornl sm mplmnt smpl ult tolrnt rrr s on pont-to-pont ommunton synronz y ntrl prossor. Ts smpl mplmntton s sunt or mns wt unrs o nos, owvr, lrly, t os not sl. To optmz t ult tolrnt rrr, w r-mplmnt t s on t ronstrut spnnn tr urn rstrt. T nw rrr lso nors t ruton squn numr, w s only n wn tr r multpl onurrnt rutons. Durn kpont/rstrt, w only us rrr to sprt rnt

4 Ckpont Tm Intrp(lnMD) Ckpont Tm Intrp(lnMD) toms 1 mllon toms toms 1 mllon toms Tm (s) Tm (ms) #ors #ors F. 3: LnMD Ckpont tm or optmzton (tm n sons) F. 4: LnMD Ckpont tm tr Optmzton (Not: tm n mllsons) pss, tr s no multpl onurrnt rutons ourrn. W oun t nw optmz rrr snntly mprovs t kpont/rstrt prormn. For xmpl, on 64K ors, t rstrt tm s rmtlly ru rom 3.6s to.15s or n 1-mllon tom systm. T xprmnt rsult s tl n Ston IV. 1.8 Ckpont Tm Krkn(Jo AMPI) Jo(9 MB/or) B. Fult Tolrn or MPI Applton on Supromputrs On ostl or monstrtn ult tolrn on MPI ppltons s tt t o sulrs on supromputrs kll t ntr o wn pross ls. To llow ny o t ult tolrn sms to work wt t o quun systms, t woul rqur moton to t o sulr to lt ult tolrnt o rovr tsl. Howvr, su n to t o sulr s not sl on toy s supromputrs. Inst, w vlop sm tt mms lur o pross wtout tully klln t. Ts ult nton sm s mplmnt s DNow() unton, w s nsrt y t usr t ny pl n tr prorm to trr lur, typlly ontroll y rnom numr nrtor. Wn t DNow() unton s ll y pross, t pross wll or to n n stop rsponn to ny ommunton s t s. T ult tton sm s mplmnt s kp-lv protool. E MPI pross pns ts uy pross prolly to norm ts uy tt t s stll lv. I tr s no rspons rom pross or rtn pro o tm, t uy pross wll nos tt t pross s, n nnoun t pross s rnk to ll otr prosss. All prosss olltvly rspon to t lur n xut t rstrt protool. A pool o som spr MPI prosss r rt t t o strtup tm to us or rstrt. Ts prosss o not ntlly run usr pplton o. Durn rstrt, on spr MPI pross wll osn to rpl t l MPI rnk to xut t usr pplton. Compr to t rl snro, t only rn n our mult ult nton sm s tt t pross os Tm (s) K 4K 8K 16K 32K #ors F. 5: Ckpont tm or Jo (on Krkn) not rlly o wy. Howvr, t rst o t ult tolrn protool s t sm s t woul ppn n rl snro. Wn o sulr s xtn to llow l pross, our sm n mmtly tk vnt o t to prov tru ult tolrn support to MPI ppltons. IV. EXPERIMENTS W vlut t ovr o pro kpontn s wll s t prormn o rstrtn ppltons tr lur. Two ppltons r us n our xprmnts. On s LnMD, molulr ynms smulton prorm wrttn n CHARM++. As typl molulr ynm smulton prorms, ts pplton s mum mmory ootprnt. T otr nmrk s Jo, 7 pont stnl MPI prorm w uss 3D omposton. It s lrr mmory ootprnt. T xprmnts r on on Intrp t Aronn Ntonl

5 5 4 Rstrt Tm Intrp(lnMD) 125 toms 1 mllon toms.2.15 Rstrt Tm Intrp(lnMD) 125 toms 1 mllon toms Tm (s) 3 2 Tm (s) #ors #ors F. 6: Rstrt tm-lnmd (Bor Optmzton) F. 7: Rstrt tm-lnmd (Atr Optmzton) Lortory, n Krkn t t Ntonl Insttut or Computtonl Sn. Intrp onssts o 163, 84 ors, 8 tryts o RAM, wt pk prormn o 557 trlops. Krkn s 9, 48 omput nos n no ontns 12 ors, 16GB o mmory, wt pk prormn o 1.17 ptlops. A. Ckpont Tm Fur 3 sows t kpont tm o LnMD or optmzton, on rom 4K to 64K ors o Intrp. Two rnt sz molulr systms r uss n t xprmnts, on wt 125, toms, n t otr wt 1 mllon toms. As n sn n t ur, ltou t kpont ovr s rltvly smll us o our n-mmory sm, t nrss lnrly s t t numr o ors nrss. Ts ws lrly u to t nssty o usn n nnt olltvs protool urn rovry. In omprson, Fur 4 llustrts t tm to kpont LnMD tr pplyn t optmztons sr n Ston II. Inst o usn pont-to-pont mplmntton o ult-wr rrr, usn t spnnn tr-s rrr rus t tm to kpont. T t o t optmzton n osrv n Fur 4. W n s tt t kpont tm rmns lmost lt wn numr o or nrss rom 4K to 64K or ot molulr systms. In prtulr, t kpont tm on 64K ors or t 1 mllon molulr systm s only out 4.2 ms. In omprson, t smulton n LnMD woul tk tns o mllsons pr tm-stp. T son nmrk s Jo prorm wrttn n MPI, w s lrr mmory prnt or kpont. In ts xprmnt, w kp t kpont sz x s 9 myts pr or, n tst t kpont tm on Krkn supromputr, wt vryn numr o ors rom 248 to 32, 768. Not tt s t numr o ors ouls, t totl mount o kpont ross t ntr systm lso ouls. Ts Jo MPI prorm runs on our ult tolrnt MPI runtm, AMPI, n uss somllo or ully utomt kpontn. T rsult s llustrt n Fur 5. As w n s tt ltou t totl kpont sz ouls vry tm wn t numr o ors ouls, t kpont tm rmns onstnt n ts tsts, n s lss tn.6 son. Ts monstrts tt our n-mmory kpont sm s ly nt, n n potntlly sl to vn lrr systms. B. Rovry Prormn W us LnMD to vlut t rstrt prormn, wt t sm two rnt sz molulr systms. Rstrt s prorm on Krkn, usn t ult nton tnqus sr n Ston III-B. T rovry tm s msur rom t tm lur s tt to t pont wr t pplton s rovr n ry to ontnu ts xuton rom t lst kpont. T rs prossor s rpl y spr prossor, n ts stt s rstor rom t kpont on t rs prossor s uy urn ts pro o tm. T rsults or t optmzton n tr r sown n Fur 6 n 7 or omprson. Sn svrl rrrs r nvolv n ts pross to nsur onsstny untl t rs prossor s rovr. By usn rrr s on trnsnt spnnn tr onstrut urn rovry, t omplxty o t rovry ovr s rs rom O(P ) to O(loP ). Bor optmzton (Fur 6), t rstrt tm nrss lmost lnrly rom.2 son on 4K ors to 3.6 son on 64K ors or ot molulr systms, wl tr optmzton, t rstrt tm s rmtlly ru, w tks only.6 son on 4K ors up to.15 son on 64K ors or t 1 mllon tom systm. Ovrll, t rstrt pross o our n-mmory ult tolrn sm s vry nt, prtly u to t ult tolrn protool w llows t pplton to rstrt rom t lst kpont n t lol mmory. It lso nts rom t t tt t pplton n rstrt wtout ull stop, so t o turn-roun tm n nw o sumsson r vo. V. RELATED WORK Tr r tr mn mtos to kpont HPC pplton: unoornt kpontn, oornt kpontn

6 n ommunton-s kpontn. In unoornt kpontn, pross npnntly svs ts stt. T nt s tt kpont n tk pl wn t s most onvnnt n tus osn t rqur synronzton to ntt kpontn. Howvr, unoornt kpontn s susptl to rollk propton, t omno t [14] w oul us systms to rollk to t nnn o t omputton, rsultn n t wst o lr mount o usul work. Gurmou t l. [15] propos n unoornt kpontn wtout omno t wt t lp o lon usul pplton msss, w s ppll to Sn- Dtrmnst MPI ppltons. Coornt kpontn rqurs prosss to oornt tr kponts n orr to orm onsstnt lol stt. Coornt kpontn smpls rovrn rom lurs us t os not sur rom rollk proptons. FTI [1] s mult-lvl oornt kpont sm usn topoloy-wr RS non wt out 8% ovr. BLCR [16] mplmnts krnl lvl kpontn n s wly us n ppltons wt prouton qulty. In [17] Mooy t l. propos multlvl kpont n us Mrkov prolty mol to sr ts prormn. On rwk or tos mtos s tt t pplton ouln t rovr n t urrnt run ust tr t lur ppns ut woul rqur t usr to rrun t pplton, rn t kpont orm t sk. Communton-nu kpontn llows t prosss to tk som o tr kponts npnntly wl prvntn t omno t y orn t prossors to tk tonl kponts s on protool-rlt normton pyk on t pplton msss t rvs rom otr prossors [18]. Howvr t s sllty ssus on lr numrs o prossors. VI. CONCLUSION AND FUTURE WORK As t sz o supromputrs nrss, t prolty o systm lur rows sustntlly, posn n nrsnly snnt lln or sllty. Ts ppr prsnt svrl optmzton tnqus to sll oul n-mmory kpont/rstrt sm to mprov ts sllty towrs xsl. W monstrt ts prormn wt mllon tom molulr ynms smulton on up to 64K ors o BluGn/P mn, n sow kpont tm n mllsons. T rstrt tms wr msur to lss tn.15 sons on 64K ors. In utur, w pln to mplmnt nonlokn kpont n our sm. For ppltons wt lr mmory ootprnt runnn on multor mns, nonlokn kpont oul urtr ovrlp kponts wt omputton n tus ru t kpont ovr. ACKNOWLEDGMENTS Ts work ws prtlly support y t US Dprtmnt o Enry unr rnt DOE DE-SC1845, y NSF rnt OCI or Blu Wtrs, n y t Insttut or Avn Computn Appltons n Tnolos (IACAT) t t Unvrsty o Illnos t Urn-Cmpn. Ts work us mn rsours rom Trr unr wr ASC539N. REFERENCES [1] L. Butst-Gomz, S. Tsuo, D. Komtts, F. Cppllo, N. Mruym, n S. Mtsuok, FTI: H prormn ult tolrn ntr or yr systms, n H Prormn Computn, Ntworkn, Stor n Anlyss (SC), 211 Intrntonl Conrn or, nov. 211, pp [2] Top5 supromputn sts, ttp://top5.or. [3] F. Cppllo, Fult tolrn n ptsl/ xsl systms: Currnt knowl, llns n rsr opportunts, Int. J. H Prorm. Comput. Appl., vol. 23, pp , Auust 29. [Onln]. Avll: ttp://l.m.or/tton.m?= [4] G. Zn, L. S, n L. V. Klé, FTC-Crm++: An In-Mmory Ckpont-Bs Fult Tolrnt Runtm or Crm++ n MPI, n 24 IEEE Intrntonl Conrn on Clustr Computn, Sn Do, CA, Sptmr 24, pp [5] G. Zn, C. Hun, n L. V. Klé, Prormn Evluton o Automt Ckpont-s Fult Tolrn or AMPI n Crm++, ACM SIGOPS Oprtn Systms Rvw: Oprtn n Runtm Systms or H-n Computn Systms, vol. 4, no. 2, Aprl 26. [6] L. V. Kl n G. Zn, Crm++ n AMPI: Aptv Runtm Strts v Mrtl Ots, n Avn Computtonl Inrstruturs or Prlll n Dstrut Appltons, M. Prsr, E. Wly-Intrsn, 29, pp [7] C. Hun, G. Zn, S. Kumr, n L. V. Klé, Prormn Evluton o Aptv MPI, n Prons o ACM SIGPLAN Symposum on Prnpls n Prt o Prlll Prormmn 26, Mr 26. [8] C. Hun, O. Lwlor, n L. V. Klé, Aptv MPI, n Prons o t 16t Intrntonl Worksop on Lnus n Complrs or Prlll Computn (LCPC 23), LNCS 2958, Coll Stton, Txs, Otor 23, pp [9] G. Zn, O. S. Lwlor, n L. V. Klé, Multpl lows o ontrol n mrtl prlll prorms, n 26 Intrntonl Conrn on Prlll Prossn Worksops (ICPPW 6). Columus, Oo: IEEE Computr Soty, Auust 26, pp [1] G. Antonu, L. Bou, n R. Nmyst, An nt n trnsprnt tr mrton sm n t P M 2 runtm systm, n Pro. 3r Worksop on Runtm Systms or Prlll Prormmn (RTSPP) Sn Jun, Purto Ro. Ltur Nots n Computr Sn Sprnr- Vrl, Aprl 1999, pp [11] G. Bronvtsky, D. J. Mrqus, K. K. Pnl, R. Run, n S. A. MK, Complr-nn nrmntl kpontn or opnmp ppltons, n Prons o t 13t ACM SIGPLAN Symposum on Prnpls n prt o prlll prormmn, sr. PPoPP 8. Nw York, NY, USA: ACM, 28, pp [Onln]. Avll: ttp://o.m.or/1.1145/ [12] J. T. Dly, A r orr stmt o t optmum kpont ntrvl or rstrt umps, Futur Gnrton Comp. Syst., vol. 22, no. 3, pp , 26. [13] J. W. Youn, A rst orr pproxmton to t optml kpont ntrvl, Commun. ACM, vol. 17, no. 9, pp , [14] B. Rnll, Systm strutur or sotwr ult tolrn, n Prons o t ntrntonl onrn on Rll sotwr. Nw York, NY, USA: ACM, 1975, pp [Onln]. Avll: ttp://o.m.or/1.1145/ [15] A. Gurmou, T. Roprs, E. Brunt, M. Snr, n F. Cppllo, Unoornt kpontn wtout omno t or sntrmnst MPI ppltons, n Prons o t 211 IEEE Intrntonl Prlll & Dstrut Prossn Symposum, sr. IPDPS 11. Wsnton, DC, USA: IEEE Computr Soty, 211, pp [Onln]. Avll: ttp://x.o.or/1.119/ipdps [16] P. H. Hrrov n J. C. Dull, Brkly l kpont/rstrt (BLCR) or Lnux lustrs, Journl o Pyss Conrn Srs, vol. 46, pp , Sp. 26. [17] A. Mooy, G. Bronvtsky, K. Moror, n B. R. Supnsk, Dsn, moln, n vluton o sll mult-lvl kpontn systm, n SC, 21, pp [18] D. Brto, A. Cuoltt, n L. Smonn, A strut omnot r rovry lortm. n Symposum on Rllty n Dstrut Sotwr n Dts Systms 84, 1984, pp