LIGO Grid Applications

Size: px

Start display at page:

Download "LIGO Grid Applications"

Noah Doyle
6 years ago
Views:

1 LIGO Grid Appications Abert Lazzarini LIGO Laboratory, Catech GriPhyN A Hands Meeting October 2003 ANL

2 LIGO Appications Deveopment Team (GriPhyN) LIGO Scientific Coaboration LIGO Laboratory UWM AEI (Germany) ISI Abert Lazzarini, LIGO Laboratory, Catech 2

LIGO and Grid Computing Reveaing the fu science content of LIGO data is a computationay and data intense chaenge Severa casses of data anaysis chaenges require arge-scae

stabe, od neutron stars (NS) ugw emission ikey to come shorty after birth of a rapidy rotating (deformed, hot) NS ugw sky is unknown usearches wi need to survey a arge

inspira chirps ) which incude spin-spin interactions wi cover a huge parameter space (~ 10 6 greater than spiness systems) uimportant for more massive systems umassive systems

3 LIGO and Grid Computing Reveaing the fu science content of LIGO data is a computationay and data intense chaenge Severa casses of data anaysis chaenges require arge-scae computationa resources usearch for gravitationa wave (GW) anaogs of eectromagnetic (EM) pusars ugw sources not ikey to have EM counterparts ufast (miisecond) EM pusars are stabe, od neutron stars (NS) ugw emission ikey to come shorty after birth of a rapidy rotating (deformed, hot) NS ugw sky is unknown usearches wi need to survey a arge parameter space ua-sky search for previousy unidentified periodic sources requires > foating point operations per second (FLOPS) ucoaescence of compact binary systems ( inspira chirps ) which incude spin-spin interactions wi cover a huge parameter space (~ 10 6 greater than spiness systems) uimportant for more massive systems umassive systems have greater GW uminosities ulikey to be the first detected uthese anayses are ideay suited for distributed (grid-based) computing Abert Lazzarini, LIGO Laboratory, Catech azz@igo.catech.edu 3

4 Efficient search of entire sky for unknown GW pusars (with no EM counterpart) requires > FLOPS # pixes on sky vs. T observation #FLOPS vs. T observation Figures taken from Brady et a., Phys. Rev. D 57, 2101 (1998) Abert Lazzarini, LIGO Laboratory, Catech azz@igo.catech.edu 4

LIGO GriPhyN Appication: Grid-depoyed production-eve GW pusar search The pusar search conducted at SC 2002 -- TOY DEMO ONLY Used LIGO data subset coected during the first scientific run of the

5 LIGO GriPhyN Appication: Grid-depoyed production-eve GW pusar search The pusar search conducted at SC TOY DEMO ONLY Used LIGO data subset coected during the first scientific run of the instrument Targeted a set of 1000 ocations incuding both known EM pusar as we as random ocations in the sky Resuts of the anaysis are avaiabe via LDAS (LIGO Data Anaysis System) DB queries Performed using LDAS and compute and storage resources at Catech, University of Southern Caifornia, University of Wisconsin Miwaukee. SC 2002 TOY DEMO ONLY Over 58 directions in the sky searched Tota of 330 tasks 469 data transfers 330 output fies produced. The tota runtime:11.4 hr Abert Lazzarini, LIGO Laboratory, Catech azz@igo.catech.edu 5

6 Goa: Use SC2003 to initiate a production anaysis run for GW pusars Stand-aone search codes to be used for the SC2003 Perform frequency-time transformations of data Parameters of transform depend on {q,f,(f, f, f, )} Code originay deveoped by GEO project as part of LIGO coaboration The code has been modified to enabe a user to specify input and output fie names and parameters on the command ine Stage data to/from compute resources as needed Auto generator wi approximatey search the gaactic core for pusars Specifies the range of parameters to be search Resuts wi be shown on a 3D Visuaization Abert Lazzarini, LIGO Laboratory, Catech azz@igo.catech.edu 6

7 Grid-depoyed production-eve GW pusar search Use SC2003 to impement production-eve scaabe version of SC2002 demonstration Origina goa: Impement a production GW pusar search over a patch (ess than 4p of sky) search running for ~30 days on O[10x] more resources than LIGO has => use the grid (10000 CPUs for 1 month) Subscae impementation using ONLY LIGO-owned resources in deveopment Joint deveopment with ISI CS team. As additiona resources become avaiabe, wi scae production run to expand beyond LIGO resources. Abert Lazzarini, LIGO Laboratory, Catech azz@igo.catech.edu 7

8 LIGO Grid Sites LIGO Grid: 6 US sites + 2 EU sites (Cardiff/UK, AEI/Germany) Coaboration: 35+ institutions word wide; 400 members Cardiff AEI/Gom Abert Lazzarini, LIGO Laboratory, Catech azz@igo.catech.edu 8

9 SC2003 production run Sites to be used ISI Condor and LSF Custers CALTECH condor poo UWM condor poo AEI (Germany) condor poo Cardiff condor poo UW Madison condor poo Teragrid resources?? Grid3 resources?? Abert Lazzarini, LIGO Laboratory, Catech 9

10 Impementation detais Web porta based authentication using MyProxy and request submission. Resource discovery and information using Gobus MDS S2 data from 3 instruments to be used. Each input fie contains data for 30 mins approximatey 25Mb in size. Executabe staging for sites which don t have geo-code instaed. Abert Lazzarini, LIGO Laboratory, Catech azz@igo.catech.edu 10

11 Impementation detais (cont.) Site monitoring being done using GANGLIA Gobus Repica Location Service used for registering raw and materiaized data products. Abert Lazzarini, LIGO Laboratory, Catech 11

12 GW Pusar Search - Dag Structure Virtua Data ExtractSFT Input transfer ComputeFstatistic Threshod & Detection Search Execution nodes Interpoo transfer Output transfer RLS Registration Abert Lazzarini, LIGO Laboratory, Catech azz@igo.catech.edu 12

13 Status First dry run week of for testing out the search code and the SC demo/production infrastructure. The jobs were run at the ISI condor poo, ISI sf custer, CALTECH condor poo and UWM condor poo The outputs transferred to the ISI condor poo The jobs breakdown is as foows 11 input transfer jobs (bringing in 1002 input fies) ( 1000 raw fies and 2 caibration fies earth.dat and sun.dat) 10 Extract SFT jobs each crunching on 100 input fies 1 ComputeFstat job which takes the 1000 extracted SFT fies and computes fstatistic. 1 interpoo transfer job which transfers data from runs at CALTECH, UWM and ISI LSF custer to the ISI condor poo (The computefstat was running at ISI LSF custer) 11 output transfers (A the output data was stored in the storage ocation associated with the ISI condor-poo) 11 RLS registration jobs ( A data was registered at rs://smarty.isi.edu associated with the ISI condor-poo) Abert Lazzarini, LIGO Laboratory, Catech azz@igo.catech.edu 13

14 GW Pusar Search - Dag Over Time Abert Lazzarini, LIGO Laboratory, Catech azz@igo.catech.edu 14

15 GW Pusar Search - Dag Over Time by Hosts Abert Lazzarini, LIGO Laboratory, Catech azz@igo.catech.edu 15

16 Run time Dag Visuaization Visuaization code by Gregor von Laszewski, and Mihae Hategan (ANL) Abert Lazzarini, LIGO Laboratory, Catech 16

17 LIGO Science Has Started First Science run ( S1 ): First LIGO scientific resuts submitted for pubication Second science run ( S2 ): Sensitivity was ~10x better than first run Duration was ~ 4x onger Third science run ( S3 ): LIGO is anayzing rea data today with emerging grid technoogies Need to baance priorities: > data anaysis chaenges vs. grid R&D chaenges Opportunity to provide case study feedback to middeware deveopment activities > robustness, QA, timeiness > tracking of opensource software reeases (e.g., inux kernes, ) > what works, what does not work, what needs improvement, Abert Lazzarini, LIGO Laboratory, Catech azz@igo.catech.edu 17

18 LIGO Appications Deveopment (outside GriPhyN) Scott Koranda GriPhyN A Hands Meeting Appications Workshop 15 October 2003 ANL G Z

19 LSC Grid Computing Appications Deveopment LSC Agorithm Library (LAL)» Forms underying data anaysis engine (Joien Creighton, Librarian) Participation in LAL deveopment effort (past 12 months)» W. Anderson, Babak, Bose, Brady, Brown, Chin, Churches, Cokear, J. Creighton, Kimenko, Krishnan, Mende, Regimbau, Sathyaprakash, Sintes, Syvestre, T. Creighton, Romano, Tibbits, Torres, Whean, Yakushin New infrastructure» Interface to frame data access Library (J. Creighton & Brown)» decimation / PSD estimation (Brown, Brady, Creighton)» LIGO Lightweight Data Format output (Brown & others as needed)» interface to caibration information (J Creighton, Brown)» improvements in fitering and injection (T Creighton)» improvements to date package (Chin) G Z

20 Prototype Grid Appication Deveopment LALApps (J. Creighton, Librarian)» Programs for desktop or grid-computing use. Binary Neutron Star Inspira Search (Brown)» Anayzed ~600 hrs S2 data in 1 weekend using Condor on 300 node custer Burst Search (Brady, Mackin, Ray-Majumdar)» Anayzed a S2 payground data in few hours using Condor on 300 node custer. Pusar (Papa, Siemens, Aen )» Aready used in S1, enhanced for S2. G Z

21 Exampe of binary inspira search DAG Inspira search code (UWM)» A tripe coincident data fitered ~ 250 hrs data/ifo» L1: 15 hrs to fiter on medusa at UWM G Z 08/21/03 1

LIGO Grid Applications Development Within GriPhyN

LIGO Grid Applications Development Within GriPhyN Albert Lazzarini LIGO Laboratory, Caltech lazz@ligo.caltech.edu GriPhyN All Hands Meeting 15-17 October 2003 ANL LIGO Applications Development Team (GriPhyN)