Stefano Cavuoti Supervisors: G. Longo 1, M. Brescia 1,2 (1) Department of Physics University Federico II Napoli (2) INAF National Institute of

Transcription

1 Stefano Cavuoti Supervisors: G. Longo 1, M. Brescia 1,2 (1) Department of Physics University Federico II Napoli (2) INAF National Institute of Astrophysics Capodimonte Astronomical Observatory- Napoli

2 Telescopes (ground-based and space-based, covering the full electromagnetic spectrum) Archive growth The ESO case Instruments(telescope/band dependent) Large digital sky surveys are becoming the dominant source of data in astronomy: ~ TB/survey (soon PB), ~ sources/survey, many wavelengths Data sets many orders of magnitude larger, more complex, and more standardized than in the past 2

3 Due to new instruments and new diagnostic tools, the information volume grows exponentially Most data will never be seen by humans! (BLADE RUNNER) The need for data storage, network, database-related technologies, standards, etc. Information complexity is also increasing greatly Most knowledge hidden behind data complexity is lost Most (all) empirical relationships known so far depend on 3 parameters. Simple universe or rather human bias? Most data (and data constructs) cannot be comprehended by humans directly! The need for data mining, KDD (Knowledge Discovery in Databases), data understanding technologies, hyper dimensional visualization, AI/Machine-assisted discovery 3

4 Semantic BoK construction PROBLEM Machine learning Models and functionalities Selected tool Catalogs and new archives RESULTS knowledge 4

5 DAME Program is a joint effort between University Federico II, Caltech and INAF-OACN, aimed at implementing (as web 2.0 applications and services) a scientific gateway for data analysis, exploration and mining, on top of a virtualized distributed computing environment. Science and management info Documents Science cases Newsletters DAMEWARE Web Application media channel 5

6 In this scenario DAME, starting from astrophysics requirements domain, has investigated the Massive Data Sets(MDS) exploration by producing a taxonomy of data mining applications (hereinafter called functionalities) and collected a set of machine learning algorithms(hereinafter called models). This association functionality-model is made of what we defined "use case", easily configurable by the user through specific tutorials. At low level, any experiment launched on the DAME framework, externally configurable through dynamical interactive web pages, is treated in a standard way, making completely transparent to the user the specific computing infrastructure used and specific data format given as input. So the user doesn t need to know anything about the computing infrastructure and almost nothing about the internal mechanisms of the chosen machine learning model.. 6

7 DR Storage DR Execution GRID SE User & Data Archives (300 TB dedicated) GRID UI GRID CE DM Models Job Execution (300 multi-core processors) Incoming DAMEWARE mirroring at Caltech Cloud facilities 16 TB 15 processors Oct 10-11, 2011 S. Cavuoti PhD Workshop 2011 AstroInformatics 2011 Sorrento, Sep DAME, M. Brescia et al. 7

8 DAta Mining Web Application REsource web-based app for massive data mining based on a suite of machine learning methods on top of a virtualized hybrid computing infrastructure Multi Layer Perceptron trained by: Back Propagation Quasi Newton Genetic Algorithm Support Vector Machines Genetic Algorithms Self Organizing Feature Maps K-Means Bayesian Networks Decision Trees next Classification Regression Clustering Multi-layer Clustering Principal Probabilistic Surfaces MLP with Levenberg-Marquardt Feature Extraction Oct 10-11, 2011 S. Cavuoti PhD Workshop 2011 AstroInformatics 2011 Sorrento, Sep DAME, M. Brescia et al. 8

9 Based on the X-Informatics paradigm, it is multi-disciplinary platform (until now X = Astro) End users can remotely exploit high computing and storage power to process massive datasets (in principle they can do data mining on their smartphone ) User can automatically plug-in his own algorithm and launch experiments through the Suite via a simple web browser Oct 10-11, 2011 S. Cavuoti PhD Workshop 2011 AstroInformatics 2011 Sorrento, Sep DAME, M. Brescia et al. 9

10 10

11 GAME (Genetic Algorithm Mining Experiment) is a pure genetic algorithm developed in order to solve supervised problems of regression or classification, able to work on Massive Data Sets (MDS) envolving just the genetic evolution (crossover, genetic mutation, roulette/tournament, elitism). GAME being a Genetic Algorithm need the creation of chromosomes population (genoma), this means that we need an internal representation (encoding genes of the chromosomes, normalization) and a fitness function able to evaluate the goodness of a chromosome than other. This depends obviusly from the problem examinated and hence from nature and the intrinsic characteristics of the dataset evaluated. In order to give a level of abstraction able to make simple adapt the algorithm to the specific problem, a family of polynomial developments was chosen. This metodology makes the algorithm itself easly expandable; this abstraction requie a set of parameters that allows to fit the algorithm to the specific problem CONVERGENCE TUBE Evaluating the goodnes of a solution with MSE or RMSE sometimes may happen that a better solution for MSE is a worse solution for the net, for example if we have a a simple classification problem with two patterns, one of class 0 and one of class 1, if the solution are 0.49 for the class 0 and 0.51 for the class 1, we have an efficiency of 100% (each patter correctly classified) and a MSE of 0.24, a solution of 0 for the class 0 and 0.49 for the class 1 (efficiency of 50%) gives back a MSE equal to 0.13 so the algorithm will prefer this kind of solution, in order to avoid this problem in GAME is implemented the convergence tube 11

12 (in coll. Con M. Paolillo & DAME coll.) The study of Globular Clusters populations in external galaxies requires the use of wide-field, multi-band photometry. In order to minimize contamination highresolution data are required BoK(TRUE CLUSTERS) selection in color Single images only one flux and morphology (reduces the cost in terms of observing time) Section of Hubble ACS image used to detect Globular Clusters around N1399. GCs (in yellow) GC are difficult to distinguish from background galaxies (in green) based only on single band images. 12

13 Results can be summarized as it follows: type of experiment missing features figure of merit MLPQNA GAME SVM MLPBP MLPGA complete patterns - class.accuracy completeness contamination no par class.accuracy completeness contamination only optical 8, 9, 10, 11 class.accuracy completeness contamination mixed 5, 8, 9, 10, 11 class.accuracy completeness contamination Classical methods bring to results about 10% worse than this 13

14 (in coll. Con M. Annunziatella& DAME coll.) Aobjectis classified astransientif itsapparent brightnessas seen from Earth changes over time, whether the changes are due to variations in the actualluminosity, or to variations in the amount of the light that is blocked from reaching Earth. The study of Transient populations in external galaxies requires the use of real-time classificators that may identify a Transient object when it s changing V838 -HST Oct 10-11, 2011 S. Cavuoti PhD Workshop

15 Oct 10-11, 2011 S. Cavuoti PhD Workshop

16 We have simulated 12 images using VST features Mag: Pixel Size: arcsec Exp Time: (s) Seeing: Image Size: 1024x1024 Yellow Stars: Cepheids Blue Flashes: Random Obj 16

17 Sky Transient Discovery Web App customizable workflow for real time classification of variable sky objects Configurable by user through web I/F; Customized mixing of third-part SW (Stuff, Skymaker, Sextractor, PSFEx, Daophot); Telescope + instrument signature setup (FOV, pixel scale, gain, readout noise ); Setup of Exp. Time, PSF model, filters, magnitude range ; Stars+galaxies+background modeling for controlled image simulation; transient models to populate simulated images; Now available Cepheids (Sandage et Tammann 2004), SN Ia (Contardo et al. 2000); Catalogue extraction to test classifiers based on machine learning models; Real images can be used for transient classification with validated models; Next steps: New transient models; Other telescope models; ML algorithms test and validation; Real data for workflow tuning; Oct 10-11, 2011 S. Cavuoti PhD Workshop AstroInformatics 2011 Sorrento, Sep DAME, M. Brescia et al.

18 Parameter Source Image Parameters Value SkyMaker MAG ZEROPOINT 26.0 ADU per sec AUREOLE RADIUS 188 Common GAIN 1.0 e-/adu SEEING 0.7 S-Extractor THRESHOLD 0.8 FILTER gauss_4.0_7x7.conv VST case #STraDiWA v1.0 1# Default configuration file Version 1.0 Simulated image # Setup Files SETUP_FILES./default.stuff,./defaultVST.sky,./default.sex #name and path of configuration files of Stuff, SkyMaker and S-Extractor # Stuff Parameters STUFF_CATALOG_NAME B.list,V.list,I.list # different for each band PASSBAND_OBS sandage/b,sandage/v,johnson/i # Observed passbands in Stuff (sandage, johnson etc are the folders contains filters) # Variable Objects TRANSIENT 1,20.3,2.5,RANDOM,RANDOM # OBJECT TYPE, INITIAL MAGNITUDE, PERIOD (days), AMPLITUDE( mag), PHASE(rad) (to have random values, RANDOM) verified by S-extractor TRANSIENT 1,21.3,1.2,1.8,2Pi # OBJECT TYPE, INITIAL MAGNITUDE, PERIOD (days), AMPLITUDE( mag), PHASE(rad) (to have random values, RANDOM)

19 PSFEx does not work directly on images. Instead, it operates on SExtractor catalogues. PSFEx_MODEL doesn t work well with the default psf provided by SExtractor. It needs a psf build on the image that we are considering. We can identify as stars the objects with PSFEx_MODEL less then a certain threshold and as galaxies the objects with PSFEx_MODEL grater equal that. The best is obtained with value 0.01, although with a little contamination for galaxies. Bin S CLASS_STAR>=0.98 /S extracted G CLASS_STAR<0.98 /G extracted mag 100,00% 100,00% Bin S PSFEx_MODEL<0.01 /S extracted G PSFEx_MODEL>=0.01 /G extracted mag 94,12% 100,00% mag 96,43% 100,00% mag 100,00% 100,00% mag 98,11% 100,00% mag 85,39% 100,00% mag 52,17% 100,00% mag 18,42% 100,00% mag 3,41% 100,00% mag 0,00% 100,00% mag 0,00% 100,00% mag 0,00% 100,00% Sextractor mag 100,00% 100,00% mag 100,00% 100,00% mag 100,00% 100,00% mag 100,00% 100,00% mag 100,00% 86,96% mag 100,00% 91,67% mag 100,00% 74,24% mag 100,00% 56,10% mag 98,17% 24,69% mag 95,04% 14,81% Sextractor -> PSFEx -> Sextractor AstroInformatics 2011 Sorrento, Sep DAME, M. Brescia et al.

20 Courses: theoretical and observational cosmology(dr. Gianpiero Mangano) 6 credits tecnologie astronomiche(dr. Massimo Brescia) 8 credits fisica delle galassie(prof. Massimo Capaccioli) 6 credits Workshops: Lecture at INGRID 2010( DAME: A Distributed Data Mining& Exploration Framework within the Virtual Observatory Lecture at 7TH International workshop Data Analysis in Astronomy, April 2011, Erice, Italy: DAME: A web oriented infrastructure for scientific data mining and exploration Local Organizing Committee: International Virtual Observatory Alliance Interoperability Workshop - Napoli, May 16-20, 2011 Local Organizing Committee: AstroInformatics 2011 Conference Sorrento, September 25-29, 2011 External Collaborations: George S. Djorgovski (CALTECH) Transient Discovery Ciro Donalek(CALTECH) Bayesian network and citizen science Ashish Mahabal(CALTECH) Transient Discovery Raffaele D'Abrusco(Harvard-Smithsonian Center for Astrophysics) AGN, Photometric redshifts Omar Laurino(Harvard-Smithsonian Center for Astrophysics) Photometric Redshifts Petr Skoda, Cech Republick, Observatory of Ondreyov Sajeet Ninaan, Poone Institute of Astrophysics, India 20

21 Refereed Publications : DAME: A distributed data mining and exploration framework within the Virtual Observatory, Brescia M., Cavuoti S., et al. 2011, in Remote Instrumentation for escience and Related Aspects, F. Davoli et al. (eds.), Springer Science+Business Media, LLC 2011, DOI / _17 DAME: A Web Oriented Infrastructure for Scientific Data Mining & Exploration, Brescia M., Longo G., Djorgovski G.S., Cavuoti S., et al., arxiv: v1 Extracting knowledge from massive astronomical data sets, Brescia M., Cavuoti S., et al., 2011, arxiv: v1, to appear in Astrostatistics and data mining in large astronomical databases, L.M. Barrosaro et al. eds, Springer Series on Astrostatistics, 15 pages The detection of globular clusters in galaxies as a data mining problem Brescia M., Cavuoti S., et al., submitted to MNRAS Conference proceedings: Astrophysics in S.Co.P.E. Brescia, M.; Cavuoti, S., et al. 2009,, Mem. S.A. It. Suppl. Vol 13, p. 56 DAME: A Web Oriented Infrastructure for Scientific Data Mining and Exploration, Cavuoti, S.; Brescia, et al., 2011,, Data Analysis in Astronomy, (accepted in press), Erice, Sicily, Italy, Apr DAME: A Distributed Web Based Framework for Knowledge Discovery in Databases, M. Brescia, G. Longo, M. Castellani, S. Cavuoti, et al., Mem. S.A.It., Vol. 75, 282, INAF OAC Napoli, Italy, 2010 DAME: A Distributed Data Mining & Exploration Framework within the Virtual Observatory, M. Brescia, S. Cavuoti, et al., INGRID 2010 Workshop on Instrumenting the GRID, Springer Editor (accepted in press), Poznan, Poland, May 12-14,

22 IN PREPARATION Machine Learning Methods for Supervised Classification in Massive Data Sets, Brescia M., Cavuoti S., Longo G., Neural Networks in preparation Data Mining in Astronomy with DAME. Part I:The Infrastructure and Services, Cavuoti S., Brescia M., Longo G. Experimental Astrophysics in preparation Data Mining in Astronomy with DAME. Part II Scientific Application Cases, Cavuoti S., Brescia M., Longo G. Experimental Astrophysics in preparation DAME: a VO compliant platform for astrophysical data mining, Brescia M., Longo G., Djorgovski G.S., Cavuoti S., et al., PASP, in preparation 22

23 DAMEWARE β release is available at PhD activities: design and implementation of other plugins and functionalities. First Results on the selection of candidate Globular Clusters in external galaxies PhD activities: tackling the second part of the work, about the X-Ray binaries selection. AGN photometric classification with machine learning methods PhD activities: tackling the classification using a spectroscopic KB. STraDiWA web app for transients real time classification PhD activities: in progress and soon we shall tackle the most challenging problem of detection and classification from simulated images and then switch to real ones. EUCLID Mission Data Quality Mining, Photo-z and Transient discovery. PhD activities: After approval by ESA (October 4), it is started the contribution in Science Ground Segment data quality for the Phase-A. We will also collaborate with Photo-z and Transients Science Teams. 23