LHC-CMS Tier2 facility at TIFR

Transcription

1 LHC-CMS Tier2 facility at TIFR T2-IN-TIFR Kajari Mazumdar Department of High Energy Physics Tata Institute of Fundamental Research Mumbai, India. NKN Workshop, IIT, Bombay November 1, 2012

2 e-science and e-research Large collaborative research is made possible by sharing resources across the globe via internet (data, computation, people s expertise...). Crosses organisational, national and international boundaries. High Energy Physics(HEP) is often compute as well as data intensive. Experiments performed by several thousand scientists over several years. CERN-LHC project is an excellent example of large scale colalboration dealing with huge amount of data. WWW was born in CERN to satisfy the needs of previous generation HEP experiments. LHC has been a driving force for GRID computing technology which evolved naturally from internet.

3 The GRID Computing Goal Science without borders. Provide Resources and Services to store/serve O(10) PB data/year Provide access to all interesting physics events to O(4000) collaborators Minimize constraints due to user localisation and resource variety Decentralize control and costs of computing infrastructure Solution through LHC Computing GRID Much faster delivery of physics

4 To begin at the end: Internal speed of computers have become comparable to the speed of large distance network. The operations of the LHC machine and the experiments have been a great success. Fantastic output, often only days after the data is taken about 200 scientific publications per experiment in 3 years. Today >140 sites ~250k CPU cores ~100 PB disk Amount of data written by LHC experiments during last 6 months: 19 Petabyte. Expected volume of data by end of 1 st phase of LHC operation: 30 PB. Total transfer rate across globe: ~ 10 Gbps LHC Computing Grid is the backbone of the success story

5 Large Hadron Collider (LHC) Largest ever scientific project 20 years to plan, build 20 years to work with 27 km circumference at 1.9 K at Torr at m below surface > 10K magnets 4 big experiments: >10K scientists, students, engineers. Operational since 2009, Q4.

6 LHC: ~ seconds (p-p) ~ 10-6 seconds (Pb-Pb) Big Bang WMAP (2001) COBE(1989) Today Experiments in Astrophysics & Cosmology ~ years

7 What happens in LHC experiment Proton-Proton bunch/beam Protons/bunch Beam energy 4 TeV / proton Luminosity 7.6*10 33 /cm 2 /s Crossing rate Total event rate 20 MHz 10 8 Hz 2 major multipurpose experiments at LHC A Toroidal LHC Apparatus (ATLAS) Compact Muon Solenoid (CMS) 10 Million electronic channels per experiment. Proton bunches collide every 50 nano sec. Higgs production <1 per sec.

8 Example of a modern detector 3170 scientists and engineers including ~800 students 169 institutes in 39 countries India

9 Data CMS as foreseen for design parameters Today s data collection rate ~ 850 Hz

10

11 Layered Structure of CMS GRID connecting computers across globe Experimental site Tier 1 National centres Several Petabytes/sec. CERN ASIA (Taiwan) Online data recording USA 10 Gbps Tier 0 Germany CERN computer centre, Geneva Italy France Tier 2 Regional groups in a continent/nation Gbps India China Korea Taiwan Pakistan Indiacms T2_IN_TIFR Different Universities, Institutes in a country TIFR BARC Delhi Univ. Panjab Univ. Individual scientist s PC,laptop,..

12 Today : 7 collaborating institutes in CMS: ~ 30 scientists + 40 students 2.1% of signing authors in publication, Contributing to computing resource of CMS ~ 3% CMS in Total: 1 Tier-0 at CERN 7 Tier-1s on 3 continents 50 Tier-2s on 4 continents CMS T2 in India : one of the 5 in Asia-Pacific region

13 Quick description of LHC grid tiers Distributed Analysis is not a wish, it is a necessity tools have to be reliable! First job of the offline system is to process and monitor data at T0. T0: 1 M jobs/day + test-jobs. traffic : 4Gbps input, > 13 Gbps served CERN Tier 0 moves ~ 1 PB data per day, automated subscription T1 processes data further several times a year, coordinates with T2s T2s are the real workhorses of the system with growing roles Hosts specific data streams for analysis Gets main data from T1s, recently more communications among T2s Typically 100k analysis jobs/day/experiment Site readiness is at high level 24X7 availability lot of invisible effort! Storage in T2 partitioned into central and local space.

14 T2-IN-TIFR, Mumbai Resources - CPU : 3000 (7000) - Disk : 700 TB (950 TB) - Nework (WAN) : Gbit/s Contributing to the computing efforts of CMS experiment credits earned against mandatory service jobs. End-user does not have to bother about the resources Middleware: - Storage Elements - Computing Elements - Workload Management - Local File Catalog - Information System A large amount of invisible human effort is essential - Virtual Organisation management - Inter-operability among different GRIDs

15 Network connections 1.5/2 Gbps to CERN peered to GEANT 2.5 Gbps NKN +TEIN3 TIFR-INDIACMS T2 VECC/SINP: INDIA 1 Gbps to VECC RRCAT, IPR

16 Monitoring the performance of a CMS T2 site Availability and reliability : fraction of time all functional tests in a site are successful Data transfers number of active links with sustained rates Job Robot (simulates small analysis jobs/user jobs) for failures, the logfile describes the nature. the investigations carried out locally + central experts.

17 ~ events produced in 2011

18 Global status

19 Data Transfers from/to TIFR upload Total data volume at present ~ 250 TB Total transfers during last 6 months ~ 70 TB download TIFR hosting i) centrally managed data (simulated, custodial) ii) collision data skims Current CMS total CPU pledge at T2s : 18k jobs slots Nominal Analysis pledge : 50% Slot utilization during Summer/Fall 09 was reasonable but need to go into sustained analysis mode

20 August 15-18, 2011 Maximum: 1.5 Gbps Avg. : 1Gbps

21 Low latency transfers

22 Physics datasets for analysis Distribution of data to participating centres all over the world Huge datasets (~ few TB) cannot be transferred to each user the analysis jobs go to where the data is Results come to user!

23 The KEY of GRID Computing: NETWORK Need a service for monitoring networks.

24 Conclusion CMS Tier2 center at TIFR is in working condition and satisfies all CMS guidelines and requirements. Making what we have today more sustainable is a challenge Need to invest MORE in networks across the country and make full use of the distributed system for a remote participation to be satisfactory! Sincere thanks to ERNET, EU-India grid and NKN for the support towards our effort in puttng India in CMS-Grid map.

25 Backup

26 Motivation of LHC experiments LHC is meant to resolve some of the most puzzling issues in physics: Nature of elementary particles and interactions shortly after Big Bang how many interactions when the universe was much hotter which elementary particles existed with what properties? we recreate conditions of very early universe at LHC. Origin of the mass mass patterns among different particles in today s universe why photon is massless, while carriers of weak interaction are massive? if the symmetry is broken spontaneously, what is the signature? existence of the God Particle? The Higgs boson yet to be discovered, but coming soon, stay tuned! LHC is at the threshold of discovery likely to change the way we are used to think of Nature!

27 Site Description 6 collaborating institutes at present, more in near future. BARC, Delhi Uni., Panjab Uni., TIFR (EHEP & HECR), VisvaBharati Uni. About 50 physicists 3.5 FTE to manage the site till now, reducing gradually. User Access to T2-IN-TIFR High end server as User Interface with the latest glite, root, CMS software versions (concurrent versions) GRID computing analysis facility using CMS-specific package CRAB Directly connected to T2 LAN Fast access to storage using RFIO 50 TB local disk space to users with individual directories AFS client, dedicated job slots (PBS) facilities Latest OS security patches

28 Grand menu from LHC Nature of dark matter: we know only 4% of the constituent of the universe A good 25% of the rest is massive enough to dictate the motion of galaxies non-luminous, and hence dark LHC can tell us the nature of this dark matter! observed LHC will also shed light on: why there is only matter and no antimatter today. properties of the 4 th state of matter: Quark-Gluon-Plasma which existed 1 pico sec. after the big bang, before formation of neutrons and protons.. Expected from visible Distribution of matter All this is possible because LHC is essentially a microscope AND a telescope as well!

29 Challenges Versatile experiments, equipped with very specialized detectors. ~10 7 electronic channels per experiment, ready every 25 ns to collect information of debris from violent collisions. Reconstruct 20K charged tracks in a single event (lead-lead collisions at LHC) 10 vertices in a single proton-proton collision, to be discriminated from the interesting process Charged tracks from heavy ion collision vertex Event size related to flux/intensity 1.5 Billion events recorded in 2010 > 2B events, much more complicated, to be recorded during 2011 resource utilization to be prioritized by carefully throwing not-so-interesting collisions.

30 Enter a New Era in Fundamental Science LHCb CMS ATLAS Exploration of a new energy frontier in p-p and Pb-Pb collisions ALICE LHC ring: 27 km circumference Discovered Higgs boson. Stay tuned!

31 Update & Upgrade on Storage Existing Storage is of ~570 TB 18 Servers : Intel Xeon E5430@2.66GHz, 4 * 1GB NIC, 16GB RAM, 24 * 1TB HDD (RAID 6), SATA 3Gb/s, 7200 RPM 4 Servers : Intel Xeon E5620@2.40GHz, 4 * 1GB NIC, 24GB RAM, 34 * 2TB HDD (RAID 6), SATA 3Gb/s, 7200 RPM 1 Server : Intel Xeon E5620@2.40GHz, 4 * 1GB NIC, 24GB RAM, 16 * 2TB HDD (RAID 6), SATA 3Gb/s, 7200 RPM Addition of 400TB storage in next 2-3 months.

32 Update & Upgrade on Managing and WN Existing Computing Resources: Computing: 360 Cores in 45 Blade Servers, Quad Core Dual processor, Intel 2.66 GHz (Clovertown), 16 GB RAM (2GB/ Core), 72GB HDD New Resources: Worker Nodes: 320 Cores in 40 Servers, Quad Core Dual processor, Intel 2.53 GHz, 24 GB ECC Registered RAM, GB HDD Managing Servers: 64 Cores in 8 Servers, Quad Core Dual processor, Intel 2.53 GHz, 24 GB ECC Registered RAM, 300 GB HDD Total CPU: 3000

33