The European Perspective for

Transcription

1 The European Perspective for Emmanuel Gangler UBP Clermont-Ferrand (France) 1/34

2 capabilities : A stage-iv survey : 8.4 m telescope Cerro Pachon (Chili) 3.2 Gpix 9.6 FoV camera 0.2 '' pixel First light 2020 All visible sky in 6 bands (ugrizy) (~20000 ) 15 s exposure, 1 visit / 3 days r ~24 / visit During 10 years! ~200 visits/band 15TB/day 60 PB/10 years 2/34

3 LSST project and Science: L3 Telescope Caméra Data Management Outreach L1 & L2 LSST covers 4 major scientifc themes Scientifc analysis is not part of the project Dark Energy, Dark matter Mapping Milky Way Transient optical sky Solar system Conducted by independent collaborations (need data rights) With the help of LSSTc 3/34

4 LSSTc institutions : Institutional members: Signifcantly contribute to LSST project and enabling science 37 institutions, 5 being non-us: Republic of Chile FR (CNRS / IN2P3) UK (Oxford & Portsmouth Universities) CZ (Institute of Physics of the Academy) European International contributors International Contributors: Support for LSST operations ( 200 k$ / PI ) 25 institutions 11 of them come from 8 European countries Last Chance Opportunity MoA to be fnalized in 2016 All have full data access rights + more coming (IT, PL?,...) 4/34

5 European interest and LSST science LSST is a world-wide project! Around 900 scientists expected to have LSST data rights ~450 from US ~300 from Europe (9 countries) arxiv:09@ /34

6 European interest and LSST science LSST is a world-wide project! Around 900 scientists expected to have LSST data rights ~450 from US ~300 from Europe (9 countries) 9 science collaborations Galaxies (46 members) 1 HR, 3 UK Stars, Milky Way, and Local Volume (118 members) 2 DE, 4 UK Solar System (N/A) Dark Energy (565 members) 2 CZ, 1 ES, 65 FR, 70 UK arxiv:09@ (Large scale structure/baryon oscillations) (Strong lensing) Active Galactic Nuclei (36 members ) 1 RS, 1 UK Transient/Variable stars (>104 members) > 2 DE, 1 FR, 1 IL, 1 UK Informatics and statistics (60 members) 1UK Those numbers are steadily growing! 6/34

7 Complementarity with ESO ESO instruments adapted for follow-up LSST data as a source for: Southern hemisphere Complete set of instruments (ex. E-ELT, 4-most) Transient alerts (2M/day) Catalog of stellar objects and galaxies (37 B objects) From ESO perspective: (ESO in the 2020 workshop) By 2020, ESO moves towards Coherent programs Time allocation to solve specifc problems in one go Importance of archives Astronomers expected to work primarily from archives Combination of archival data and new observations Statistical astronomy 7/34

8 Complementarity with ESA programs LSST is a natural successor of GAIA LSST and EUCLID share 5000 of the sky Stage IV LSST Starts, duration Area (deg2) FoV (deg2) Diameter (m) Spectroscopic Survey 2022, 10yr 20,000 (S) Photometry 6 bands (ugrizy) Euclid WFIRST 2020 Q2, 6.25yr ~2025, 5-6 yr 15,000 (N + S) 2,400 (S) Grism R=250 Grism R= mm mm one broad optical 3 NIR bands + IR (JHK) Sensitivity improved by combining results Joint analysis can be key in some areas combination at the data level 8/34

9 Maximizing the use of LSST data Complementarity with ESO/ESA projects From NOAO/LSST recommendations (arxiv ) Need for follow-up instruments (wide-feld MOS on a 8m, OIR spectroscopy, multicolor imaging) But before analyzing data, we need A working project to acquire data (start in 2020) An effcient way to enable data access (15PB catalog...) 9/34

10 Challenges : building LSST L3 Telescope Caméra Data Management Outreach L1 & L2 10/34

11 Challenges : building LSST L3 Telescope Caméra Data Management Outreach L1 & L2 Construction site on september 9th /34

12 Camera placeholder if time Sensor tests ComCam Sizeable hardware from FR Partner of LSST project since /34

13 Data management is a pillar of the project : L3 Telescope Caméra Data Management Outreach L1 & L2 «The data volumes [ ] of LSST are so large that the limitation on our ability to do science isn't the ability to collect the data, it's the ability to understand [ ] the data» Andrew Conolly (U. Washington) How do you turn petabytes of data into scientific knowledge? Kirk Borne (George Mason U.) 13/34

14 How to handle astronomical big data? LSST is strongly inspired by Astroinformatics point of view : Borne /34

15 How to handle astronomical big data? LSST is strongly inspired by Astroinformatics point of view : Raw data L1 & L2 Data products + User tables L3 & science output Borne /34

16 How to handle astronomical big data? LSST is strongly inspired by Astroinformatics point of view : Raw data L1 & L2 Data products + User tables L3 & science output Borne /34

17 From Raw data to L1 & L2 products Image Processing pipeline by Data Management team Validation on precursor data (SDSS Stripe 82, CFHTLS, HSC) 17/34

18 From Raw data to L1 & L2 products EU expertise on image reduction algorithms (ex. CFHTLS,DES...) Brighter-Fatter effect Low surface brightness Guyonnet 2015 PSF broadens with the flux strong impact for Weak Lensing!! Slide from Cebrián Requires excellent control of PSF and sky 18/34

19 From Raw data to L1 & L2 products Embarrassingly parallel problem suited for HTC computing (I/O bound) Per MoA, FR will: Process 50% of data up to L2 proucts Host an archive of LSST data There will be a full copy of LSST data in EU 19/34

20 Towards Data Access Center(s?) in Europe FR will host a full copy of LSST data This is a real opportunity to set up something ambitious We have to think about the next step DAC infrastructure has a cost Investment, Running cost/year Options under discussion: UK gets ready for a DAC center (Edinburgh) FR will host a DAC at least for its own needs (Lyon) Focussed on relevant scientifc topics (DESC) Coupled to analysis facility Extension to serve the broader European LSST community Requires coordination at the EU level A European DAC can be multisite... 20/34

21 Available data: Alerts : 2 M/day (within 60s.) Static: 80 TB image In 6 bands (all visible sky) Dynamic: 60 PB Relation 1-Many Object Sources Catalogs : 15 PB 21/34

22 Bokering LSST Alerts Proposed System: Preparting for the data fow: L1 data products Alert Generation (image difference /catalog/etc. ) Validation (Real/Bogus) Based on simulation Alertsim False positive will leak through Miss false negative Format for alert distribution Event Brokers Annotation Filtering Ranking VO compliant 2-4 brokers Community observers Follow-up Classifcation From Jeremović Feedback needed 22/34

23 Distributing LSST data EU Contributors: The baseline Orchestration tool Partitioning : SQL parser Metadata DB User defned function (geometry) Communication with xrootd MySQL Backend Returns agregate results Limitations Geometry (cone searches) Sources and Object in the same node SQL-based Some queries can't be treated (time, volume) Ad hoc optimization 23/34

24 Lessons learned: Qserv works well! Better than specs Data deployment is not trivial Network confguration Need 2-3x more disk space Previous tests : 300 nodes Large results stress master node Each scale increase raises new issues test platforms are vital 24/34

25 Alternate approaches: Map/Reduce Qserv Still ftp-grep model Will fail when selectivity is low! Better use of local CPU Spherical geometry can be worked out Indexing required Limitations: Map lacks global knowledge Reduce lacks data access Calculus (Cartoon from PhD comics) Query DBMS x Controler How effciciently can we move the calculus to the data? Interdisciplinary research! 25/34

26 Map-reduce-like implementation on Qserv How to use local CPU? Tuple-at-a-time functions (ex: B-V) Trivial processing Well-behaved aggregation functions This includes a very broad class of operations (sums, averages, linear least-squares,...) Local node : f(xi) = (q1, q2, q3,...)= Q_j Global node : Res = a(q_j) With local storage of intermediate results, can even be included in a more complex workfow Can be implemented by minimal tweaking of Qserv This kind of interdisciplinary research can be conducted with small academic grant 26/34

27 Which knowledge to extract? and how to extract it? This is what info-astronomers do already!??? Raw data L3 L1 & L2 Data products & science output + User tables??? Borne /34

28 Example 1 : photo-z Photo-z are critical for LSST Which classifer? Performance issue Systematic uncertainty Training set Small spectroscopic sample 50M spectra, 20 B galaxies Different coverage in parameter space Photo-z

29 Example 2 : supernovae Photometric classifcation Which features to extract? time domain offers new challenges Which classifer? Performance issue Systematic uncertainty due to false positive Training set Small spectroscopic sample Only 10% of LSST SN will have spectroscopic confrmation Too small non-ia sample See also E. Ishida talk, this conf. Lochner 2016

30 New ways to extract knowledge.. and how to apply it to astro data? This is what computer scientists do! New method Raw data L1 & L2 Data products??? + User tables New method Borne /34

31 From a Computer Scientist perspective LSST data are interesting! Non-commercial source of Big Data and in a database! Offers challenges: Machine-learning feature selection (e.g. Deep Learning) Inference from incomplete reference data set (Outlier/novelty detection, Master Data Management,... ) Uncertainties How to improve astronomical knowledge from new methods? Data Mining Machine Learning Effcient algorithmic for Front lines, Cliques, Rules discovery Training! interdisciplinary approach! (for European Computer Sciences too)

32 Getting organized Europe is a great place to work on LSST Funding opportunities (Training, Networking, Infrastructure) COST BigSkyEarth : 25 countries Interdisciplinary: Computer Sciences, Astrophysics, Earth Observation Data Access, Data Anaysis, Data Visualization, Training Next conference: october 24th-25th in Sorrento 32/34

33 Getting organized Europe is a great place to work on LSST Interest of LSST for Europe LSST@Europe conference series Edition 2016 in Belgrade (included Astroinformatics session) Next edition in France (2018) Topical workshops Getting ready for doing science with LSST data (Lyon, 2017) organized with the LSST DM team 33/34

34 Conclusions LSST will provide unprecedented astronomical data Addresses major science topics: ~300 European participants from 9 (10 with IT) countries Synergies with European facilities (ESA, ESO,...) Sizeable investment from European researchers and institutions LSST Camera Construction Archive, Processing and Data centers getting ready for analyzing LSST data in Europe New challenges for informatics: Both for database experts and data mining (+visualization) community Lot of expertise in EU Only 1 European member in LSST informatics and statistics group are we missing something? 34/34

35 35/34

36 LSST Time Domain data Main survey: ~200 visits / feld / band ~1 visit/3 days Over 10 years Depth as uniform as possible Actual cadence being optimized Roughly uniform coverage But specifc needs Revisit time (SN, NEOs) Proper motions (Emphasis year 1 and 10) Signifcant improvements possible Input from community highly welcomed

37 ~ 1/ of LSST data! LSST data: Impressive data fux : Images : 2x6.4 Gbyte/39 seconds 15 TB/night (data rate 330 MB/s) 0,5 EB of produced data objects ( 100 à 200 TB ) «sources» detection ( 3 à 5 PB ) «forced sources» measurements ( 1 à 2 PB) >2.106 transient alerts / night Simulation 1 CCD 4k x 4k 37/34

38