protostelle, classe 0, caratterizzate da un'intensa emissione alle lunghezze della radiazione submillimetrica, che però diviene molto debole a λ<10 µm

Transcription

1 Data Mining

2

3 protostelle, classe 0, caratterizzate da un'intensa emissione alle lunghezze della radiazione submillimetrica, che però diviene molto debole a λ<10 µm Le sorgenti di classe I hanno spettri la cui intensità aumenta molto rapidamente al crescere della lunghezza d'onda λ e irradiano maggiormente a λ>20 µm; le sorgenti di classe II hanno uno spettro molto più piatto, con contributi quasi uguali nel vicino e nel lontano infrarosso; le sorgenti di classe III possiedono uno spettro che irradia maggiormente per λ<2 µm e si affievolisce nettamente per λ>5 µm.

4 Examples of star nursery regions Bottom M17, right Magellan cloud

5 Spectral energy distributions (SEDs) describing the energy objects that radiate over the entire electromagnetic spectrum have conventionally been the main way of classifying the age and evolutionary phase of young stellar objects (YSOs). YSOs attain most of their mass when an isolated rotating dense core collapses and forms an accreting protostellar core and disc that accumulates surrounding interstellar gas and dust via large gravitational forces. Recent theoretical studies suggest that SED information alone is not sufficient to accurately determine whether YSOs are early (Class 0) or late (Classes I, II and III) phase. Rather, a combination of SED and near-infrared (NIR) spectroscopy that enables characterisation of the accreting protostar are required.

6 An evolutionary classification tool for ViaLactea, will catalogue clumps in terms of the evolutionary stage and mass regime of the ongoing star formation. There are two components that need to be developed at the foundation of the classification tool: 1. an evolutionary classification toolbox 2. a set of star-forming clumps in known stages of evolution to be used as a training/test-set for machine-learning algorithms...and adopt some kind of evolutionary scheme Data-mining approaches to source classification Unsupervised Analysis We know nothing about the sources evolutionary stage; Identify over-densities in the given parameter space (e.g., built on the evolutionary toolbox, plus any other available evidence); Data are then grouped into clusters: groups of data entries sharing common but a priori unknown correlations among parameter space features. Weak Gated Classification For a subsample of points, its category/class is treated as bayesian inference; Need order of 10 3 known objects to be used as a training set; Balanced population of classes in the training set.

7 Luminosity/mass diagram IR color distribution Mass/radius relationship Association with masers Association with radio continuum Outflow indicators YSO IR-color spatial association Others? ambiguity SF clumps source class prediction Quiescent / pre-stellar Proto-stellar Hot molecular cores Ultra-compact HII regions Evolved HII regions classe 0 classe I classe II classe III

8 Luminosity/mass diagram IR color distribution Mass/radius relationship Association with masers Association with radio continuum Outflow indicators YSO IR-color spatial association Others? ambiguity SF clumps source class prediction Quiescent / pre-stellar Proto-stellar Hot molecular cores Ultra-compact HII regions Evolved HII regions

9 Luminosity/mass diagram IR color distribution Mass/radius relationship Association with masers Association with radio continuum Outflow indicators YSO IR-color spatial association Others? ambiguity Quiescent / pre-stellar SF clumps source class prediction Knowledge Base Proto-stellar Hot molecular cores Ultra-compact HII regions Evolved HII regions Train set Validation set Test set Fuzzy/cross-entropy

10 Given: S=Evolutionary stage (pre-stellar, proto-stellar, hot-cores, HII,.) cluster y=parameter space coordinates (luminosity, temperature, colour, ) clump features Bayes Theorem p S y = p(s) p y S p(y)

11 Given: S=Evolutionary stage (pre-stellar, proto-stellar, hot-cores, HII,.) cluster y=parameter space coordinates (luminosity, temperature, colour, ) clump features Bayes Theorem p S y = p(s) p y S p(y) Probability for the object having y to belong to S OUR TARGET

12 Given: S=Evolutionary stage (pre-stellar, proto-stellar, hot-cores, HII,.) cluster y=parameter space coordinates (luminosity, temperature, colour, ) clump features Bayes Theorem p S y = p(s) p y S p(y) Probability to have the evolutionary stage S BY SCIENTIFIC EXPERTS

13 Given: S=Evolutionary stage (pre-stellar, proto-stellar, hot-cores, HII,.) cluster y=parameter space coordinates (luminosity, temperature, colour, ) clump features Bayes Theorem p S y = p(s) p y S p(y) Normalization parameter

14 Given: S=Evolutionary stage (pre-stellar, proto-stellar, hot-cores, HII,.) cluster y=parameter space coordinates (luminosity, temperature, colour, ) clump features Bayes Theorem p S y = p(s) p y S p(y) Probability for an object in the evolutionary stage S to have y LIKELIHOOD STELLAR MODELS

15 HI-GAL clumps data are not sufficient to have a parameter space carrying useful and exhaustive information The analysis of compact sources belonging to a clump could give a first set of information (features) useful to increase the dimensions of the parameter space to explore ClumpPopulator (ASSOCIATION CLUMPS STELLAR SOURCES) Multi-threading software based on the positional cross-match among sources at different wavelengths STEvMSCARS (STELLAR SOURCES CLASSIFICATION) Multi-threading software based on prescriptions characterizing the color-color and color-magnitude diagrams

16 GLAT The positional cross-match at the basis of the ClumpPopulator method is based on the same concept of the Q-FULLTREE approach: Definition of an elliptical region centred on galactic coordinates of clump and semi-axes defined by the FWHM values Search for higher resolution sources within such region x 2 a 2 + y2 b 2 1 GLON At each clump will be therefore associated a number of sources falling in the corresponding ellipse. Green ellipse: Clump Green dot: Clump center Red dot: NIR Source

17 Besides the association clump-source in the elliptical region with the FWHMs as semi-axes, it is possible to extend the association to a user-defined number of additional ellipses, concentric to the basic clump ellipse and with gradually increasing and/or decreasing dimensions This is useful in order to evaluate, in a post-processing phase, the stellar density in the clump compared with the external region to search clump over-densities

18 1) Clumps output Clumps data from clumps catalogue list of the sources associated to the clump number of the sources associated to the clump For each concentric ellipse dimensions and the number of associated sources 2) Sources output Sources data from the input sources catalogue list of the clumps associated to the source number of the clumps associated to the source This file is related only to the original ellipse representing the clump and not to the additional concentric ellipses

19 GLAT I-Remover: INTERSECTION REMOVER Some clumps can have one or more common associated sources. If the user does not want to use such clumps in further analysis, he can use this tool to delete intersecting clumps from the knowledge base. GLON Green ellipse: Clump Green dot: Clump center Red dot: NIR Source OUTPUT: ClumpPopulator output without entries related to the intersecting clumps.

20 SDE: STELLAR DENSITY EVALUATOR In order to obtain information about the evolutionary stage of a clump and to characterize it by means of the associated sources found, it can be useful to understand if some clumps show any over-density occurrence. SDE provides information about the source density of a clump and the areas defined by the concentric ellipses evaluated by ClumpPopulator. In particular it provides the density of sources/unit area within the elliptical rings obtained as difference between two consecutive ellipses and some related statistical quantities Yellow: Basic Ellipse of clump Red: Concentric Ellipses Blue dots: NIR Source Density of sources in i-th ring (Nsources i Nsources i 1 )/A ring i Additional rings are added IN and OUT of the clump original ellipse

21 STEvMSCARS (Software Tool on Evolutionary Model for Stars and Clumps Analysis, Research and Statistics) is a software designed and developed to characterize a clump in terms of the evolutionary stage of its compact sources, associated to the clump by ClumpPopulator A method to estimate the evolutionary stage of a sky object is to study its position in the color color and magnitude color diagrams, where to identify specific ROIs correlated with the evolutionary stage. Each region of a diagram can be represented by a set of functions, prescriptions, defining its physical limits. Each prescription is indeed composed by a set of analytical rules (usually inequalities), engaging relations among different magnitudes and colors of the compact sources under investigation. Current available prescriptions have been taken from literature

22 Input Catalog Prescriptions implemented as Python functions and applied recursively on all the sources of an input catalogue. Flagging System Source Prescription Flagging Pre-STEvMSCARS Prescriptions Current version of STEvMSCARS and its prescriptions are tailored on GLIMPSE, UKIDSS and MIPS data Input: magnitudes (and/or colours) of each source. Output: at each source is associated, for each available prescription: a binary flag indicating if the related prescription has been satisfied (0=false; 1=true); source-prescription distance, minimum distance of the source in the diagram from each function composing a prescription; source-prescription normalized distance, source-prescription distance normalized by the maximum distance of all the sources with the same flag

23 Pre-STEvMSCARS Output Columns designation ID of sources sources data of the input catalogue (not shown) for each available prescription assigned flag Distance normalized distance

24 Source/Clump file association file files Clumps Statistics EVO-STATS Source/Clump association Prescriptions file file Flagging Classified Sources Classification and a statistical analysis on the sources belonging to the clumps of the Hi-GAL catalogue, by combining the outcome from both ClumpPopulator and Pre-STEvMSCARS. 1. Each source is classified as contaminant, YSO class I, II or III object; 2. Given the catalogue of clumps, it counts how many sources belong to a certain class are in each clump, taking into account the clumpsources association obtained by ClumpPopulator. Input: Pre-STEvMSCARS output (even from multiple catalogues) Clump-sources association by ClumpPopulator (even from multiple catalogues) Output: list of sources classified as contaminant, YSO class I, II or III list of the clumps and, for each clump the counts of source objects belonging to each involved class.

25 Flagged Source from Pre-STEvMSCARS IRAC + NIR Statistical Indicators Evaluation Contaminant? IRAC Yes CONTAMINANT CANDIDATE No NIR Statistical Indicators Evaluation No YSO I or II? IRAC + NIR Yes Can distinguish? Yes YSO I CANDIDATE No YSO II CANDIDATE YSO III? UNDEFINED MIPS No UNCLASSIFIED Yes YSO III FINAL CLASS (YSO I, II, Contaminant) Re-examination YSO I II Contaminants YSO I Refinement Transition Disk (YSO I) Embedded protostar (YSO II)

26 Clumps Statistics CSV file containing the following information for each clump: Clump Data (from clumps catalogue file) Counts of sources (satisfying each prescription) Counts of sources (satisfying each refinement prescription) Number of classified sources (for each class) Number of unclassified and undefined sources Number and List of sources (in the clump) Classified Sources CSV file containing the following information for each source: Statistical Indicators Value (IRAC+NIR) Candidate Class (before refinement) Reliability Flag (indicating if the classification is well univocally defined) Prescription Flags (for each refinement prescription) Final Class Flag and Bitwise Statistical Indicators and Candidate Class (only IRAC) Statistical Indicators and Candidate Class (only NIR)

27 Multithread python: STEvMSCARS code is optimized in order to minimize computational time by using a multithreading approach Easily updatable with new prescriptions The user can easily design and codify personal prescriptions by using few code lines and following few simple steps It works with arbitrary sources catalogues and bands The possibility to create new prescription implies that STEvMSCARS can be used on arbitrary sources catalogues and bands, that must satisfy only few (and standard) requirements EVO-STATS can use multiple Pre-STEvMSCARS results It is possible to use output of the first phase applied simultaneously on different catalogues (for example on a cross-matched catalogue UKIDSS GLIMPSE and a catalogue of UKIDSS sources without IRAC counterparts), clearly after having preliminarily applied the ClumpPopulator tool to all involved catalogues

28 We have tested STEvMSCARS method, in collaboration with M. Merello (IAPS), on three different regions and compared the results with ones reported in literature Region Reference N sources IRAS W51 W49 Yun et al Kang et el 2009 Saral et al 2015 Data description 377 Spitzer + NOTCam 104,582 Glimpse I + 2MASS + RIASSUNTO RISULTATI MIPS STEVMSCARS 332,442 Glimpse + 2MASS/UKI DSS+MIPS Note Contaminants not considered IRAC classification based on spectral index, not directly comparable Classification based on the procedure by Gutermuth et al 2008 GENERAL RESULTS (mainly based on comparison with Saral et al. 2015): Good agreement with other works; STEvMSCARS recovers and flags properly the contaminant sources; The recovery of YSOs (as a whole) works fine when photometry on all IRAC and JHK bands is available. Differences are found when specific classes (I and II) are needed; Very good agreement (95%) for sources not considered as YSO by STEvMSCARS; Differences with classification by using NIR data.

29 An evolutionary classification tool for ViaLactea, will catalogue clumps in terms of the evolutionary stage and mass regime of the ongoing star formation. There are two components that need to be developed at the foundation of the classification tool: 1. an evolutionary classification toolbox 2. a set of star-forming clumps in known stages of evolution to be used as a training/test-set for machine-learning algorithms...and adopt some kind of evolutionary scheme Available Data UKIDSS GPS Survey GLIMPSE Surveys (I II 360 3D Deep Vela Carina) HIGAL Catalogues (Single Band, Bandmerged) HIGAL Physical Properties Catalogue CORNISH Radio Continuum Catalogue

30 SURVEY DATA GLON RANGE (deg) GLAT RANGE (deg) UKIDSS HIGAL GLIMPSEI GLIMPESII GLIMPSE360 GPS DR10PLUS Multimission Catalogue 2007 Catalogue (FINAL) Spring 08 Catalogue (FINAL) FINAL 141< l <230 15< l <107-2< l <15 0 l l < l <65 295< l <350 5< l <10 2< l <5 0< l <2 350< l < < l < < l < < l <102.4 {[76,82]} 65< l <265 {[76,82],[102,109]} b <5 b <5 b <2-1.87<b<1.79 b <1 b <1 b <1.5 b <2 b <1 b <1.5 b <2 b <3 GLIMPSE3D FINAL Only vertical extension GLIMPSEI and GLIMPSEII b >1 265< l <295-2<b<0.1 DEEP GLIMPSE DR2 25< l <65 0<b<2.7 VELA CARINA DR1 255< l < <b<1.5 UKIDSS GLIMPSE HIGAL Intersection: GALACTIC LONGITUDE: 0 l 60 GALACTIC LATITUDE: -1 b 1

31 Example: Sub-region GLON: 50 l 60, GLAT: -1 b 1 Distance: 0.5 Type: Best Matches UKIDSS GLIMPSE Crossmatch Hi-GAL Multimission Hi-GAL Properites Merging Sub-region Extraction Sub-region Extraction Distance: max(obs_size) Type: All Matches UKIDSS GLIMPSE Hi-GAL Crossmatch 50 l 60-1 b 1 Separation(i) OBS_SIZE(i) Confining SOURCES in CLUMPS UKIDSS GLIMPSE Hi-GAL DATASET TO DO: Addition of Radio Continuum information (CORNISH)

32 COUNTS GLAT Green circle: Clump Green dot: Clump center Red dot: NIR Source GLON NIR sources in 4798 Hi-GAL clumps (~6 sources/clump) clumps without UKIDSS-GLIMPSE counterpart (~7%) <1 [1,3] [4,6] [7,10] [11,15] [16,20] [21,30] >30 NUMBER OF SOURCES IN A CLUMP

33 J-H H-K Color Color Diagrams: J-H vs H-K H-K vs K-[4.5] Reddening and Extinction estimation Add 24µm Band (MIPSGAL) H-K References: Megeath S. T. et al., 2012, ApJ, 144, 192 Flaherty K.M. et al., 2007, ApJ, 663, 1069 Lucas P. W. et al., 2008, MNRAS, 391, 136 K-[4.5]

34 We can estimate the evolutionary stage of a clump, by studying the SED of the sources belonging to it (MIPSGAL at 24µm for slope evaluation) Characterization of clumps by their constituent sources Contaminants Sources Detection Relationship between NIR Sources and Clumps properties Lada, 1987 and with the new Bandmerged Catalog?

35 The good preliminary results obtained by STEvMSCARS encouraged us to test some ML methods in order to learn the hidden cross-correlations among the source parameters and to classify sources properly, especially when they are not covered in all bands. Knowledge Base definition based on the classification results obtained by STEvMSCARS on dataset of Saral et al. 2015

36 3 Test campaigns: 1. Train of a classifier through data with 7 bands (IRAC+NIR), to reproduce the classification through prescriptions (in order to validate the expected capabilities of classifiers, by matching the prescriptions); 2. Train of a classifier through data with 4 bands (only IRAC), to evaluate the capability to classify the blind test set, in particular the YSO III objects (in principle not classificable with IRAC bands only); 3. Train of a classifier through data with 3 bands (only NIR), to evaluate the capability to classify the blind test set, in particular the Contaminant objects (in principle not classificable with NIR bands only). Starting Whole Catalogue Region W49 l ϵ [ , ] b ϵ [ , ] Reference Saral et al 2015 Number of Objects 332,442 Contaminants 4,318 YSO I + YSO II (YSO) 4,867 YSOIII 211,825

37 Test 1 7 bands Test 2 4 bands Test 3 3 bands Experiment Type Cont Dataset composition YSOI+ YSOII YSOIII Train (80%) Test (20%) Classifiers MLPQNA 0,01 0,1 Random Forest (YSOI+YSOII) vs Contaminants 806 1,069 1, yes no yes (YSOI+YSOII) vs YSOIII 1, , yes no yes (YSOI+YSOII) vs Contaminants vs YSOIII 806 1, , yes yes yes (YSOI+YSOII) vs Contaminants 924 1,183 1, yes no yes (YSOI+YSOII) vs YSOIII 1,183 1,001 1, yes no yes Contaminants vs YSOIII 924 1,001 1, yes no yes (YSOI+YSOII) vs Contaminants vs YSOIII 806 1, , yes no no (YSOI+YSOII) vs Contaminants 806 1,069 1, yes yes no (YSOI+YSOII) vs YSOIII 1, , yes no no (YSOI+YSOII) vs Contaminants vs YSOIII 806 1, , yes no no Two different learning decays (best cases after pruning heuristics)

38 IRAC+NIR (YSO I+II) vs CONT (YSO I+II) vs YSOIII (YSO I+II) vs CONT vs YSOIII MODEL MLPQNA RF MLPQNA RF MLPQNA RF DATA % TRAIN OBJs 1,500 1,576 2,220 TEST OBJs QNA DECAY MEAN EFFICIENCY YSO PURITY CONT PURITY YSOIII PURITY YSO COMPLETENESS CONT COMPLETENESS YSOIII COMPLETENESS YSO CONTAMINATION CONT CONTAMINATION YSOIII CONTAMINATION MLPQNA reaches strong classification agreement with prescriptions (very good balancing between purity and completeness) Remark: the percentages could easily increase by extending/refining the knowledge base

39 IRAC (YSO I+II) vs CONT (YSO I+II) vs YSOIII CONT vs YSOIII (YSO I+II) vs CONT vs YSOIII MODEL MLPQNA RF MLPQNA RF MLPQNA RF MLPQNA MLPQNA DATA % TRAIN OBJs 1,685 1,747 1,540 2,220 TEST OBJs QNA DECAY MEAN EFFICIENCY YSO PURITY CONT PURITY YSOIII PURITY YSO COMPLETENESS CONT COMPLETENESS YSOIII COMPLETENESS YSO CONTAMINATION CONT CONTAMINATION YSOIII CONTAMINATION Classifiers show affordable performances to separate YSOIII from YSOI+II, which in principle would be very difficult with only IRAC information. Results could be improved by increasing data amount (both quantity and density distribution).

40 NIR (YSO I+II) vs CONT (YSO I+II) vs YSOIII (YSO I+II) vs CONT vs YSOIII DATA % MODEL MLPQNA MLPQNA MLPQNA MLPQNA TRAIN OBJs 1,500 1,576 2,220 TEST OBJs QNA DECAY MEAN EFFICIENCY YSO PURITY CONT PURITY YSOIII PURITY YSO COMPLETENESS CONT COMPLETENESS YSOIII COMPLETENESS YSO CONTAMINATION CONT CONTAMINATION YSOIII CONTAMINATION Classifiers show affordable performances to separate Contaminants from YSOI+II, which in principle would be very difficult with only NIR information. Results could be improved by increasing data amount (both quantity and density distribution).

41 Sources Classification by STEvMSCARS Existent prescriptions optimization; New prescriptions implementation (for example, WISE); Classification schema optimization (optimal thresholding, statistical optimization, e.g. ROC curves, K-S test); Classification comparison in other regions/with other works (for example, Lada 1987). Machine Learning approach Test with different classifiers (SVM, LEMON); Augmenting data amount and parameter space coverage (density distribution); Fine tuning of model hyper-parameters; Analysis of combined mis-classification; Mid-term Goal: to obtain an extended catalogue of not yet classified sources

42 IR1, IR2, IR3, IR4 + NIR1, NIR2, NIR3 catalogue1 IR1, IR2, IR3 + NIR1, NIR2, NIR3 4 IRAC bands 3 NIR bands IR1, IR2, IR4 + NIR1, NIR2, NIR3 IR1 + NIR1, NIR2, NIR3 IR1 + NIR1, NIR2 IR1 + NIR1, NIR3 MASTER catalogue cataloguen The Master catalogue will extend the starting Base of Knowledge to be used for clump classification Final goal: classification of Hi-GAL clumps through member classification

43 Several further fields of galactic astrophysics have not been explored, due to lack of time and resources. Nevertheless they are quite appealing in terms of data exploration: Band-merging optimization: current solutions based only on topological information and cross-matching. It could be investigated the integration with information related to fluxes and dimensions of sources, with the goal to confirm/discard the fitness of candidate SEDs found. Dimensions: to evaluate the standard deviation between a well-posed sample of known sources and new ones. The result may label sources that need to be reprocessed by reduction pipelines to refine their dimensions. Fluxes: they could be analyzed in terms of color-color diagrams among available bands and their MSE, trying to find out of trends events, requiring to be reprocessed and refined. Statistical tests: K-S or t-student statistical analyses could be used to compare SED model libraries with candidate SEDs, trying to find best fitting occurrences. Post-merging optimization: from current output (trees of candidate SEDs) further merit scores, based on flux/dimension analysis, could refine the score given from topological cross-match. Such merit score functions should be thinked and tested.

44 500 CSS 1 CL = 0.91 CL = 0.87 CL = 0.89 NE TNE = 3 3 = 1 MS 1 = 2.67 MS 2 < MS 1 MS 2 = CSS 2 CL = 0.91 CL = 0.87 CL = CL = 0.02 NE TNE = 4 6 = 2 3

45 CSS CSS 2

46 Several further fields of galactic astrophysics have not been explored, due to lack of time and resources. Nevertheless they are quite appealing in terms of data exploration: Distance estimation: incremental KB (starting from masers and well-resolved band-merged sources) used as training for supervised methods to predict source distances. The parameter space could include: spectral information from data cube slices (like CO, H1, H12 etc.) density maps, fluxes and colors from available bands and so on Evolutionary Classification: current solutions provide statistics based on clump density dimensions. The idea would be to grow up the RoIs to improve statistics. There are several parameters that could be used as: Input: already available from catalogues there are L/M, dust temp., surface density, luminosity, mass, efficient radius; Post-analysis: filament-clump association, maser-clump association, radio counterparts (CORNISH); Such parameters could be used to perform unsupervised analysis (clustering) but driven by the knowledge on a small subset of well-known sources. Extinction Maps: the image processing based on data mining tools can be used to detect and reconstruct the maps of galactic extinction directly from the images.

47 Data Mining is defined as an information extraction activity whose goal is to discover hidden facts contained in (large) databases The overall goal of the data mining process is to extract information from a data set and transform it into an understandable structure for further use It is an heuristic process that requires long time and it is not sure that the final result will be as expected, even if also a bad result can give important information on how to build or modify the parameter space It also requires a continuously interaction (from the parameter space definition to the validation of results) between science and astroinformatics experts and, eventually, a Knowledge Base in order to apply some Machine Learning approach This kind of work cannot be started and finished in the three years of the ViaLactea project The work started in the ViaLactea project is only the starting point in order that Data Mining become a standard method in the astronomical research