The New LSST Informatics and Statistical Sciences Collaboration Team. Kirk Borne Dept of Computational & Data Sciences GMU

Transcription

1 The New LSST Informatics and Statistical Sciences Collaboration Team Kirk Borne Dept of Computational & Data Sciences GMU

2 We are facing a huge problem! The Tsunami

3 We are facing a huge problem! The Data Tsunami

4 Data Doubles Every Year Computing power doubles every 18 months (Moore s Law) x in 10 years I/O bandwidth increases ~10% / year <3x in 10 years. Data doubles every year x in 10 years, and 1,000,000x in 20 yrs. NCSA Example: First 19 years: 1 PB Year 20 (2007): 2 PB Year 21 (2008): 4 PB By 2020 : ~20 Exabytes? In the Year 2525 : PB????? As our data volumes grow, especially in the sciences (where scientific funding for research barely grows at all), we will fall farther and farther behind in our ability to analyze, assimilate, and extract knowledge from our data collections... unless we develop and apply exponentially more powerful algorithms and methods. (Illustration adapted from a slide by J. Heer, PARC User Interface Research Group) 4

5

6 National Study Groups Face the Data Flood Several national study groups have issued reports on the urgency of establishing scientific and educational programs to face the data flood challenges. Each of these reports has issued a call to action in response to the data avalanche in science, engineering, and the global scholarly environment.

7 Data Sciences: A National Imperative 1. National Academies report: Bits of Power: Issues in Global Access to Scientific Data, (1997) downloaded from 2. NSF (National Science Foundation) report: Knowledge Lost in Information: Research Directions for Digital Libraries, (2003) downloaded from 3. NSF report: Cyberinfrastructure for Environmental Research and Education, (2003) downloaded from 4. NSB (National Science Board) report: Long-lived Digital Data Collections: Enabling Research and Education in the 21st Century, (2005) downloaded from 5. NSF report with the Computing Research Association: Cyberinfrastructure for Education and Learning for the Future: A Vision and Research Agenda, (2005) downloaded from 6. NSF Atkins Report: Revolutionizing Science & Engineering Through Cyberinfrastructure: Report of the NSF Blue-Ribbon Advisory Panel on Cyberinfrastructure, (2005) downloaded from 7. NSF report: The Role of Academic Libraries in the Digital Data Universe, (2006) downloaded from 8. National Research Council, National Academies Press report: Learning to Think Spatially, (2006) downloaded from 9. NSF report: Cyberinfrastructure Vision for 21st Century Discovery, (2007) downloaded from JISC/NSF Workshop report on Data-Driven Science & Repositories, (2007) downloaded from DOE report: Visualization and Knowledge Discovery: Report from the DOE/ASCR Workshop on Visual Analysis and Data Exploration at Extreme Scale, (2007) downloaded from DOE report: Mathematics for Analysis of Petascale Data Workshop Report, (2008) downloaded from NSTC Interagency Working Group on Digital Data report: Harnessing the Power of Digital Data for Science and Society, (2009) downloaded from National Academies report: Ensuring the Integrity, Accessibility, and Stewardship of Research Data in the Digital Age, (2009) downloaded from

8 New Book! The Fourth Paradigm: Data-Intensive Scientific Discovery The 4 Scientific Paradigms: 1. Experiment (sensors) 2. Theory (modeling) 3. Simulation (HPC) 4. Data Exploration (KDD)

9 The Scientific Data Flood Scientific Data Flood Large Science Project Pipeline

10 Have you noticed the H 2 O theme? Data Tsunami Data Flood Data Deluge Data Firehose NSF s OOI : Ocean Observatories Initiative

11 Recent (March 2010) NSF working group: Data-Enabled Science (DES) DES group prepared a white paper of related challenges and recommendations to inform the NSF MPSAC (Mathematical & Physical Sciences directorate Advisory Committee). MPSAC may use the white paper as a source of ideas and information to advise NSF in DES areas. DES committee: 2 scientists each from Astronomy, Physics, Chemistry, Mathematics, Statistics, and Materials Science.

12 Examples of recommendations in the DES report to the MPSAC Research: Provide enhanced research opportunities in DES, with specific MPS tracking mechanisms to insure DES research is adequately supported and represented in science portfolios Place greater emphasis (and budget-level guards) on data management & analysis in large projects Have dedicated DES review panels at the divisional level and possibly the directorate level (e.g., astro has galaxies, stars panels) Education, Training, & Workforce development: CAREER awards in DES research areas Fellowship programs (graduate and postdoctoral) e.g., new NSF program: Fellowships for Transformative Computational Science using CyberInfrastructure (CI TraCS) Provide REU supplements and other educational initiatives in DES

13 Astronomy!

14 Astronomy is a Discovery-driven Science Discoveries lead to : New questions New ideas New models New theory Data And most importantly... New data! Discoveries have shown that the astronomical zoo is rich and diverse... Black Holes Quasars Supernovae Pulsars Blazars Tidal Streams Colliding Galaxies Magnetars Gamma-ray bursts Exo-planets Brown Dwarfs Serendipity!! Gravitational Lenses Incoming Killer Asteroid

15 From Data-Driven to Data-Intensive Astronomy has always been a data-driven science It is now a data-intensive science: Astroinformatics And it will become even more data-intensive in the coming decade(s) Some key data-driven questions for astronomers: What is it? Where is it? What causes that behavior? When did it form? How did it form? Why did it do that? Who will let me use their telescope to get more data???

16 The Changing Landscape of Astronomical Research Past: 100 s to 1000 s of independent distributed heterogeneous data/metadata/information repositories. Today: Astronomical data are now accessible uniformly from federated distributed heterogeneous sources = the Virtual Observatory. Future: Astronomy is and will become even more data-intensive in the coming decade with the growth of massive data-producing sky surveys. Astroinformatics (data-intensive astronomical research) will become a stand-alone scientific research discipline (similar to Bioinformatics, Geoinformatics, Cheminformatics, and many others). Informatics is the discipline of organizing, accessing, mining, & analyzing information describing complex systems (e.g., the human genome, or Earth, or the Universe). X-informatics is a key enabler of scientific discovery in the era of dataintensive science. (X = Bio, Geo, Astro,...) (Jim Gray, KDD-2003) Astroinformatics (intelligent data discovery, browse, integration, mining, and visualization research tools) will enable exponential knowledge discovery within exponentially growing data collections.

17 Astronomy Data Environment: Sky Surveys To avoid biases caused by limited samples, astronomers now study the sky systematically = Sky Surveys Surveys are used to measure and collect data from all objects that are contained in large regions of the sky, in a systematic, controlled, repeatable fashion. These surveys include (... this is just a subset): MACHO and related surveys for dark matter objects: ~ 1 Terabyte Digitized Palomar Sky Survey: 3 Terabytes 2MASS (2-Micron All-Sky Survey): 10 Terabytes GALEX (ultraviolet all-sky survey): 30 Terabytes Sloan Digital Sky Survey (1/4 of the sky): 40 Terabytes and this one is just starting: Pan-STARRS: 40 Petabytes! Leading up to the big survey next decade: LSST (Large Synoptic Survey Telescope): 100 Petabytes!

18 Sky Surveys: Partly the Solution and partly the Problem As our chemistry friends say... If you are not part of the solution, then you are part of the precipitate!

19 The VO and Large Surveys: What more do we need? VO (Virtual Observatory): provides the infrastructure and protocols for data discovery & access DM / ML (Data Mining & Machine Learning) tools are part of application layer, which is just beginning to blossom... The VO is essential, but alone it is not enough! Large Surveys: lots of data, with lots of project introspection lots of project self-focus on the "other DM" = Data Management. The focus on big data should morph into focusing on big science (enabling petascale query processing, data manipulation, information retrieval, data mining, knowledge discovery, information visualization)... Large Surveys are essential they provide a tonnabytes, but alone with VO they are not enough! Informatics / ML / AI / Data Science approach: the 4 th leg of science the three most important aspects are metadata, metadata, and metadata for integration, self-learning, recommendations, self-discovery, relevance analysis, dimension reduction, feature selection, constraint-based mining, semantic mark-up, knowledge mining, knowledge-reuse, and more We need more work here!

20 The VO and Large Surveys: What more do we need? VO (Virtual Observatory): provides the infrastructure and protocols for data discovery & access DM / ML (Data Mining & Machine Learning) tools are part of application layer, which is just beginning to blossom... The VO is essential, but alone it is not enough! Large Surveys: lots of data, with lots of project introspection lots of project self-focus on the "other DM" = Data Management. The focus on big data should morph into focusing on big science (enabling petascale query processing, data manipulation, information retrieval, data mining, knowledge discovery, information visualization)... Large Surveys are essential they provide a tonnabytes, but alone with VO they are not enough! Informatics / ML / AI / Data Science approach: the 4 th leg of science the three most important aspects are metadata, metadata, and metadata for integration, self-learning, recommendations, self-discovery, relevance analysis, dimension reduction, feature selection, constraint-based mining, semantic mark-up, knowledge mining, knowledge-reuse, and more... We need more work here!

21 Data-Enabled Science: Scientific KDD (Knowledge Discovery from Data) Characterize the known (clustering, unsupervised learning) Assign the new (classification, supervised learning) Discover the unknown (outlier detection, semi-supervised learning) Graphic from S. G. Djorgovski Benefits of very large datasets: best statistical analysis of typical events automated search for rare events

22 3 Categories of Scientific Knowledge Class Discovery Discovery from Data Clustering of scientific parameters Finding new classes of objects and behaviors Novelty Discovery Finding new, rare, one-in-a-million(billion)(trillion) objects and events Correlation Discovery Finding new patterns and dependencies, which reveal new physics or new scientific principles

23 Class Discovery: feature separation and discrimination of classes across multiple databases Reference: The separation of classes improves when attributes from disparate databases are chosen to be projected, as in the following star-galaxy discrimination test: Not good Good IVOA Interop Baltimore October

24 Novelty Discovery (Outlier Detection): improved discovery of rare objects across multiple databases IVOA Interop Baltimore October

25 % of variance captured by PC1+PC2 Correlation Discovery: Fundamental Plane for 156,000 cross-matched Sloan+2MASS Elliptical Galaxies: plot shows variance captured by first 2 Principal Components as a function of local galaxy density. Reference: Borne, Dutta, Giannella, Kargupta, & Griffin 2008 Slide Content Slide content Slide content Slide content IVOA Interop Baltimore October 2008 low (Local Galaxy Density) high 25 10

26 LSST = Large Synoptic Survey Telescope (mirror funded by private donors) 8.4-meter diameter primary mirror = 10 square degrees! Hello! (design, construction, and operations of telescope, observatory, and data system: NSF) (camera: DOE)

27 This is the real LSST mirror (August 2008), which will be figured and polished over the next 6 years

28 This is the real LSST mirror (August 2008), which will be figured and polished over the next 6 years (notice the cool guy in the middle of the back row in the gold LSU tigers shirt)

29 LSST Key Science Drivers: Mapping the Universe Solar System Map (moving objects, NEOs, asteroids: census & tracking) Nature of Dark Energy (distant supernovae, weak lensing, cosmology) Optical transients (of all kinds, with alert notifications within 60 seconds) Galactic Structure (proper motions, stellar populations, star streams) LSST in time and space: When? Where? Cerro Pachon, Chile Model of LSST Observatory

30 Observing Strategy: One pair of images every 40 seconds for each spot on the sky, then continue across the sky continuously every night for 10 years ( ), with time domain sampling in log(time) intervals (to capture dynamic range of transients). LSST (Large Synoptic Survey Telescope): Ten-year time series imaging of the night sky mapping the Universe! 100,000 events each night anything that goes bump in the night! Cosmic Cinematography! The New Education and Public Outreach have been an integral and key feature of the project since the beginning the EPO program includes formal Ed, informal Ed, Citizen Science projects, and Science Centers / Planetaria.

31 The LSST focal plane array Camera Specs: (pending funding from the DOE) x4096 pixels each! = 3 Gigapixels = 6 GB per image, covering 10 sq.degrees = ~3000 times the area of one Hubble Telescope image LSST Data Challenges Obtain one 6-GB sky image in 15 seconds Process that image in 5 seconds Obtain & process another co-located image for science validation within 20 s (= 15-second exposure + 5-second processing & slew) Process the 100 million sources in each image pair, catalog all sources, and generate worldwide alerts within 60 seconds (e.g., incoming killer asteroid) Generate 100,000 alerts per night (VOEvent messages) Obtain 2000 images per night Produce ~30 Terabytes per night Move the data from South America to US daily Repeat this every day for 10 years ( ) Provide rapid DB access to worldwide community: Petabyte image archive Petabyte database catalog 31

32 LSST Summary Plan (pending Decadal Survey): commissioning in Gigapixel camera One 6-Gigabyte image every 20 seconds 30 Terabytes every night for 10 years 100-Petabyte final image data archive anticipated all data are public!!! 20-Petabyte final database catalog anticipated Real-Time Event Mining: 10, ,000 events per night, every night, for 10 yrs Follow-up observations required to classify these Repeat images of the entire night sky every 3 nights: Celestial Cinematography

33 The LSST Data Challenge Massive data stream: ~2 Terabytes of image data per hour that must be mined in real time (for 10 years). Massive 20-Petabyte database: more than 50 billion objects need to be classified, and most will be monitored for important variations in real time. Massive event stream: knowledge extraction in real time for 100,000 events each night.

34 What is an Event? Anything that changes (motion or brightness) Variable stars of all kinds Optical transients: e.g., extra-solar planets Supernova Gamma-ray burst New comet New asteroid Incoming Killer Asteroid Anything that goes bump in the night

35 Here is an example of an event *** Optical Transient: here today, gone tomorrow It is a normal dwarf star, similar to our sun, except it increased in brightness by 300x in one night and then returned to normal. ***Courtesy: Caltech / DPOSS

36 Palomar-Quest VOEventNet: a Rapid-Response Telescope Grid PQ next-day pipelines VOEventNet baseline sky Raptor GRB satellites catalog Palomar 60 PQ Event Factory Event Synthesis Engine VOEvent database VOEventNet Pairitel known Variables known asteroids 2MASS SDSS remote archives estar Reference:

37 MIPS = MIPS model for Event Follow-up Measurement Inference Prediction Steering Heterogeneous Telescope Network = Global Network of Sensors: Similar projects in NASA, Earth Science, DOE, NOAA, Homeland Security, NSF DDDAS (voeventnet.org, skyalert.org) Machine Learning enables IP part of MIPS: Autonomous (or semi-autonomous) Classification Intelligent Data Understanding Rule-based Model-based Neural Networks Temporal Data Mining (Predictive Analytics) Markov Models Bayes Inference Engines

38 Example: The Thinking Telescope Reference:

39 From Sensors to Sense From Data to Knowledge: from sensors to sense (semantics) Data Information Knowledge

40 The LSST will represent a 100,000-fold increase in the VOEvent network traffic and in the real-time classification demands: from data to knowledge! from sensors to sense! See some ideas about how to handle this Event flood through a semantic-based Astronomy Data Annotation System (AstroDAS) portal described here :

41 The LSST Data Challenges

42 XLDB: an approach to petascale databases XLDB = extremely Large Databases Since 2007: 3 XLDB Workshops and 1 working meeting XLDB4 conference: October 5-7, 2010 at Stanford/SLAC The result is a new design for petabyte-scale scientific databases = SciDB SciDB is based on the new column-based data model Relational data model (RDBMS) is so last century Column-store data is the latest hot fad in databases e.g., check out and check out C-Store, which was described in this VLDB research paper: References: XLDB on Wikipedia: XLDB SLAC: SciDB: Column-oriented DB: 43

43 The LSST Petascale Challenges (document is available on-line)

44 Example: Petascale Computational & Data Science Challenge problem for LSST Find the optimal simultaneous solution for 20,000,000,000 objects shapes across 2000 image planes, each of which has 201x4096x4096 pixels floating-point operations! This illustrates an example for just one such object: References:

45 How can scientists become involved? Use the public data from LSST, with the rest of the world, after the survey has begun (~2017) University can petition the LSST Board of Directors to become an institutional member of the LSST Consortium This includes annual member dues (non-trivial) Some hard and some soft benefits (listed later) e.g., **trivial application process to join any collaboration Join a science collaboration team Must write a proposal, except for a special case**

46 How can scientists become involved? Use the public data from LSST, with the rest of the world, after the survey has begun (~2017) University can petition the LSST Board of Directors to become an institutional member of the LSST Consortium This includes annual member dues (non-trivial) Some hard and some soft benefits (listed later) e.g., **trivial application process to join any collaboration Join a science collaboration team Must write a proposal, except for a special case**

47 32 LSSTC Institutional Members Brookhaven National Laboratory California Institute of Technology Carnegie Mellon University Chilé Cornell University Drexel University [ your university here ] Google Inc. Harvard-Smithsonian Center for Astrophysics Institut de Physique Nucléaire et de Physique des Particules (IN2P3) Johns Hopkins University Kavli Institute for Particle Astrophysics and Cosmology at Stanford University Las Cumbres Observatory Global Telescope Network, Inc. Lawrence Livermore National Laboratory Los Alamos National Laboratory National Optical Astronomy Observatory Princeton University Purdue University Research Corporation for Science Advancement Rutgers University SLAC National Accelerator Laboratory Space Telescope Science Institute Texas A & M University The Pennsylvania State University The University of Arizona University of California, Davis University of California, Irvine University of Illinois at Urbana-Champaign University of Michigan University of Pennsylvania University of Pittsburgh University of Washington Vanderbilt University

48 Mason is planning to join this consortium of universities!

49 Benefits to Mason on becoming an Institutional Member of the LSST Consortium Mason can participate in LSSTC board meetings Mason can nominate a member to the voting board Publicity (e.g., in press releases and documents) Trivial application process to join any collaboration Early access to LSST pre-cursor data and commissioning data, before the survey becomes operational Gain familiarity with data and data systems (access, DB) Higher level of service from data centers (in terms of allocated CPU cycles, data transfer bandwidth, data downloads) Some (?) benefit on LSST-related science proposals

50 How can scientists become involved? Use the public data from LSST, with the rest of the world, after the survey has begun (~2017) University can petition the LSST Board of Directors to become an institutional member of the LSST Consortium This includes annual member dues (non-trivial) Some hard and some soft benefits (listed later) e.g., **trivial application process to join any collaboration Join a science collaboration team Must write a proposal, except for a special case**

51 LSST Science Collaborations Original 10 teams: 1. Solar System 2. Milky Way & the Local Volume 3. Transients/Variable Stars 4. Stellar Populations 5. Supernovae 6. Galaxies 7. Active Galactic Nuclei (AGN, quasars) 8. Strong Lensing 9. Weak Lensing 10. Large-scale Structure / Baryon Oscillations

52 What do science collaborations do? Examine and elaborate science use cases Specify and quantify science requirements Validate that the LSST data archive and databases (schema and interfaces) will support science use cases & meet requirements Do pre-cursor science projects to prepare for LSST e.g., conduct mini-sky surveys to collect multi-wavelength data in special areas, locate interesting deep drilling fields, test observing cadences for discovery potential, develop new algorithms (such as photometric redshifts, transient event classification, multi-fit 1000 s of images) Write science documents, including The LSST Science Book (600 arxiv: ) Work with Data Management team to stress-test the system Attend LSST All Hands Meetings: advise, inform, be informed

53 We have said to the LSST project managers: one research team is missing We have stressed to the LSST project for a couple of years that there is one more research area that is not represented in the 10 teams. That area is Data Mining Research. The Computer Science (data mining and machine learning) research community is already aware of and interested in LSST (astronomy data are abundant, interesting, and free). This community wants to do research with LSST. Similarly, the statistics community has discovered interesting astronomy data and the LSST.

54 So we wrote a proposal to the LSSTC: to form a new research collaboration team Our proposal was approved it was the only such proposal for a new research collaboration team that they have ever approved!

55 LSST Science Collaborations Original 10 teams: 1. Solar System 2. Milky Way & the Local Volume 3. Transients/Variable Stars 4. Stellar Populations 5. Supernovae 6. Galaxies 7. Active Galactic Nuclei (AGN, quasars) 8. Strong Lensing 9. Weak Lensing 10. Large-scale Structure / Baryon Oscillations Now one more: the ISSC

56 The new Mason-led ISSC research team In discussing the data-related research challenges posed by LSST, we identified several research areas: Statistics Data & Information Visualization Data mining (machine learning) Data-intensive computing & analysis Large-scale scientific data management Informatics These areas represent Statistics and the science of Informatics (Astroinformatics) = Data-intensive Science = the 4 th Paradigm of Scientific Research Hence, ISSC was formed = the Informatics and Statistical Sciences Collaboration

57 The LSST ISSC Research Team Chairperson: K.Borne, GMU Core team: 3 astronomers + 2 = K.Borne (scientific data mining in astronomy) Eric Feigelsen, Tom Loredo (astrostatistics) Jogesh Babu (statistics) Alex Gray (computer science, data mining) Full team: ~30 scientists ~60% astronomers ~30% statisticians ~10% data mining, machine learning computer scientists Original ISSC proposal: 50+ co-signers, only half were astronomers

58 LSST ISSC Team Members Jogesh Babu Pennsylvania State University Matthew Graham Caltech Andrew Ptak NASA/GSFC Kirk Borne George Mason University Alexander Gray Georgia Institute of Technology Jeffrey Scargle NASA /Ames Research Center Robert Brunner University of Illinois Carlo Graziani University of Chicago Chad Schafer Carnegie Mellon University Tamas Budavari Johns Hopkins University Douglas Burke Smithsonian Astrophysical Observatory Jon Hakkila College of Charleston Vinay L. Kashyap SAO Aneta Siemiginowska Harvard Center for Astrophysics Keivan Stassun Vanderbilt University David Chernoff Cornell University Kevin Knuth University at Albany Martin Weinberg Univ. of Massachusetts Jim Cordes Cornell University Tom Loredo Cornell University Roy Williams Caltech George Djorgovski Caltech Ashish Mahabal Caltech Robert Wolpert Duke University Eric Feigelson Pennsylvania State University Peter Freeman Carnegie Mellon University Christopher Genovese Carnegie Mellon U Bruce McCollum Caltech Chris Miller University of Michigan Misha Pesenson Caltech Michael Woodroofe Statistics Department, University of Michigan

59 Some key astronomy problems that require informatics and data science techniques Probabilistic Cross-Matching of objects from different catalogues The distance problem (e.g., Photometric Redshift estimators) Star-Galaxy Separation Cosmic-Ray Detection in images Supernova Detection and Classification Morphological Classification (galaxies, AGN, gravitational lenses,...) Class and Subclass Discovery (brown dwarfs, methane dwarfs,...) Dimension Reduction = Correlation Discovery Learning Rules for improved classifiers Classification of massive data streams Real-time Classification of Astronomical Events Clustering of massive data collections Novelty, Anomaly, Outlier Detection in massive databases

60 Informatics-based Science Education Informatics enables transparent reuse and analysis of scientific data in inquiry-based classroom learning ( Students are trained: to access large distributed data repositories to conduct meaningful scientific inquiries into the data to mine and analyze the data to make data-driven scientific discoveries The 21 st century workforce demands training and skills in these areas, as all agencies, businesses, and disciplines are becoming flooded with data. Numerous Data Sciences programs now starting at several universities (GMU, Caltech, RPI, Vanderbilt, Michigan, Cornell, ). CODATA ADMIRE initiative: Advanced Data Methods and Information technologies for Research and Education

61 Data Science paper available!

62 Astroinformatics paper available! Addresses the data science challenges, research agenda, application areas, use cases, and recommendations for the new science of Astroinformatics

63 Summary: Mason s Once & Future Role in the LSST Project Senior participant on EPO (Education & Public Outreach) team, including Citizen Science Participation in Science Data Management Chair of the new LSST ISSC Research Team Soon CDS + P&A depts will send application to become LSST Consortium Institutional Member Rich spectrum of opportunities for multi-disciplinary research projects for faculty and students across departments: computational & data sciences, astronomy, statistics, data mining & machine learning, visualization, science & math (STEM) education