The Case for Provenance Juande Santander-Vela, Arancha Delgado European Southern Observatory, Archive Department

Size: px

Start display at page:

Download "The Case for Provenance Juande Santander-Vela, Arancha Delgado European Southern Observatory, Archive Department"

Randell Carter
6 years ago
Views:

1 IVOA InterOp Meeting Garching, 09/11/09 The Case for Provenance Juande Santander-Vela, Arancha Delgado European Southern Observatory, Archive Department

2 Talk Outline What is Provenance? How does Provenance fit in the VO? The case(s) for Provenance Conclusions

3 What is Provenance?

4 What is Provenance? provenance noun the place of origin or earliest known history of something : an orange rug of Iranian provenance. the beginning of something's existence; something's origin : they try to understand the whole universe, its provenance and fate. See note at origin. a record of ownership of a work of art or an antique, used as a guide to authenticity or quality : the manuscript has a distinguished provenance. ORIGIN late 18th cent.: from French, from the verb provenir come or stem from, from Latin provenire, from pro- forth + venire come.

5 What is Provenance? provenance noun the place of origin or earliest known history of something : an orange rug of Iranian provenance. the beginning of something's existence; something's origin : they try to understand the whole universe, its provenance and fate. See note at origin. a record of ownership of a work of art or an antique, used as a guide to authenticity or quality : the manuscript has a distinguished provenance. ORIGIN late 18th cent.: from French, from the verb provenir come or stem from, from Latin provenire, from pro- forth + venire come.

6 What is Provenance? provenance noun the place of origin or earliest known history of something : an orange rug of Iranian provenance. the beginning of something's existence; something's origin : they try to understand the whole universe, its provenance and fate. See note at origin. a record of ownership of a work of art or an antique, used as a guide to authenticity or quality : the manuscript has a distinguished provenance. ORIGIN late 18th cent.: from French, from the verb provenir come or stem from, from Latin provenire, from pro- forth + venire come.

7 What is Provenance? provenance noun the place of origin or earliest known history of something : an orange rug of Iranian provenance. the beginning of something's existence; something's origin : they try to understand the whole universe, its provenance and fate. See note at origin. a record of ownership of a work of art or an antique, used as a guide to authenticity or quality : the manuscript has a distinguished provenance. ORIGIN late 18th cent.: from French, from the verb provenir come or stem from, from Latin provenire, from pro- forth + venire come.

8 What is Provenance? Definition points out to: History of astronomical data products Ownership/Attribution (Observer, Proposal, Telescope/Instrument) Quality

9 What is Provenance?

10 What is Provenance? TYPICAL ASTRONOMICAL WORKFLOW List of targets Find existing data per target and band Filter out data regarding quality parameters Manipulate retrieved datasets Data provenance information needed

11 What is Provenance? TYPICAL ASTRONOMICAL WORKFLOW List of targets Find existing data per target and band CHARACTERISATION Filter out data regarding quality parameters Manipulate retrieved datasets Data provenance information needed

12 What is Provenance? TYPICAL ASTRONOMICAL WORKFLOW List of targets Find existing data per target and band CHARACTERISATION Filter out data regarding quality parameters Manipulate retrieved datasets Data provenance information needed

13 What is Provenance? raw files uses knowledge of data products F 1 P 1 Astronomical Source Galactic & Extragalactic extinction Atmosphere (Seeing & Opacity) F 2 Data Reduction Software P 2 F n P m photon emission Telescope, Filters, Detectors data generation

14 What is Provenance? raw files uses knowledge of data products F 1 P 1 Astronomical Source Galactic & Extragalactic extinction Atmosphere (Seeing & Opacity) F 2 Data Reduction Software P 2 F n P m photon emission Telescope, Filters, Detectors data generation

15 What is Provenance? raw files uses knowledge of data products F 1 P 1 Astronomical Source Galactic & Extragalactic extinction Atmosphere (Seeing & Opacity) F 2 Data Reduction Software P 2 F n P m photon emission data generation Telescope, Filters, Detectors NOT ONLY FITS FILES

16 What is Provenance? raw files uses knowledge of data products F 1 P 1 Astronomical Source Galactic & Extragalactic extinction Atmosphere (Seeing & Opacity) F 2 Data Reduction Software P 2 F n P m photon emission data generation Telescope, Filters, Detectors NOT ONLY FITS FILES ALSO CATALOGUES

17 What is Provenance? Project Metadata Observation Observation Provenance Provenance Processing Ambient Instrumental Software Calibration INTERNAL PROVENANCE WHITEPAPER BY N. DELMOTTE

18 What is Provenance? ORGANISATIONAL Project Metadata Observation Observation Provenance Provenance Processing Ambient Instrumental Software Calibration INTERNAL PROVENANCE WHITEPAPER BY N. DELMOTTE

19 What is Provenance? ORGANISATIONAL Project Metadata Observation Observation ACQUISITIONAL Provenance Provenance Processing Ambient Instrumental Software Calibration INTERNAL PROVENANCE WHITEPAPER BY N. DELMOTTE

20 How does Provenance fit in the VO?

21 How does Provenance fit in the VO? raw files uses knowledge of data products F 1 P 1 Astronomical Source Galactic & Extragalactic extinction Atmosphere (Seeing & Opacity) F 2 Data Reduction Software P 2 F n P m photon emission Telescope, Filters, Detectors data generation

22 How does Provenance fit in the VO? raw files uses knowledge of data products F 1 P 1 Astronomical Source Galactic & Extragalactic extinction Atmosphere (Seeing & Opacity) F 2 Data Reduction Software P 2 F n P m photon emission Telescope, Filters, Detectors data generation

23 How does Provenance fit in the VO?

24 How does Provenance fit in the VO? RADAMS Data Model for Observation Observation Curation ObsData Policy Characterization Provenance Target Packaging Coverage Resolution Precision Sensitivity Accuracy Data Model for Dataset Characterisation

25 How does Provenance fit in the VO? RADAMS Ph.D. Thesis Data Model for Observation Observation Curation ObsData Policy Characterization Provenance Target Packaging Coverage Resolution Precision Sensitivity Accuracy Data Model for Dataset Characterisation

26 How does Provenance fit in the VO? Extended Provenance RADAMS Ph.D. Thesis Data Model for Observation Observation Curation ObsData Policy Characterization Provenance Target Packaging Coverage Resolution Precision Sensitivity Accuracy Data Model for Dataset Characterisation

27 How does Provenance fit in the VO? RADAMS Ph.D. Thesis Data Model for Observation Observation Curation ObsData Policy Characterization Provenance Target Packaging Coverage Resolution Precision Sensitivity Data Model for Dataset Characterisation Accuracy SEE TALK BY FRANÇOIS BONNAREL

28 The case(s) for Provenance

29 The case(s) for Provenance Use cases for Provenance, centred around the core concepts: History of a data set (documentation) Establishing ownership/providing attribution Quality assessment

30 History of a data set (ESO)

31 Level 0 Period History of a data set (ESO) Program Type RunID Level 3 Level 4 Program ID Project bibcode ProgID Level 1 Level 2 DataID Run ID Run ID Run ID Data Release ID Night Night Structured ReadMe Association IDs Night Log OB ID Reduced Science Data Master Calib. Data Ancillary Main File(s) Associated File(s) Raw File (DP_ID) File DP_ID DP_ID DPR CATG

32 History of a data set (ESO) ambient filters observations ID telescope instrument technique mode ambient seeing air mass extinction telescope comm/decomm efficiencies observatory name instruments instruments comm/decomm name telescope / focus technique modes detectors optical elements mappings technique comm/decomm modes mappings detector comm/decomm sensitivity modes comm/decomm total transmission curve name instrument optical elements mappings ID name PWL FWHM (type) comm/decomm transmission curve mappings grisms ID name wavelength range grating blaze angle dispersion resolution comm/decomm transmission curve mappings gratings ID name orders wavelengths... gap order dispersion resolution comm/decomm mappings slits COMPILATION BY A. DELGADO ID name slit width type comm/decomm mappings etc.

33 History of a data set (ESO) ambient filters observations ID telescope instrument technique mode ambient seeing air mass extinction telescope comm/decomm efficiencies observatory name instruments instruments comm/decomm name telescope / focus technique modes detectors optical elements mappings technique comm/decomm modes mappings detector comm/decomm sensitivity modes comm/decomm total transmission curve name instrument optical elements mappings ID name PWL FWHM (type) comm/decomm transmission curve mappings grisms ID name wavelength range grating blaze angle dispersion resolution comm/decomm transmission curve mappings gratings ID name orders wavelengths... gap order dispersion resolution comm/decomm mappings slits COMPILATION BY A. DELGADO ID name slit width type comm/decomm mappings etc.

34 History of a data set (ESO) filters grisms observatory gratings telescope instrument technique mode detector slits ambient objectives etc. COMPILATION BY A. DELGADO

35 History of a data set (ESO) filters grisms observatory gratings telescope instrument technique mode detector slits ambient objectives etc. raw files uses knowledge of data products F 1 P 1 COMPILATION BY A. DELGADO Astronomical Source photon emission Galactic & Extragalactic extinction Atmosphere (Seeing & Opacity) Telescope, Filters, Detectors F 2 F n Data Reduction Software P 2 P m data generation

36 History of a data set (ESO) COMPILATION BY A. DELGADO Telescope Information Site Dates: First Light, Commissioning Instruments & Optical Elements Instruments Technologies Configurations Optical Elements Installation dates Kind (Filter, Grisms, Gratings ) Names IDs Transmission Curves

37 History of a data set (ESO) Telescope Information Site Dates: First Light, Commissioning Instruments & Optical Elements Instruments Technologies Configurations Optical Elements Installation dates Kind (Filter, Grisms, Gratings ) COMPILATION BY A. DELGADO TIME CHANGING Names IDs Transmission Curves

38 History of a data set Provenance Observation Location +Instrument.locationName latitude -longitude -height Instrument -name -description -shortname -locationname -URL InstrumentConf AntennaConf Feed BeamConf Antenna -name -polarisation: L,R,X,Y -name +AntennaConf.name -mount -majoraxis -minoraxis -effectivearea RADAMS This set is defined for each instrument band AND receiver configuration { Beam +beamconf.name -beammajor -beamminor -sensitivity -meanbeamsolidangle -minorbeamsolidangle -directivity -gain -resolution Receiver +beamconf.name -skycentrefreq -ifcentrefreq -bandwidth Spectrum +beamconf.name -chanseparation -freqresol -velrefframe -refchannum -refchanfreq -restfreq -molecule -transition Velocity +beamconf.name -chanseparation -velresol -velrefframe -refchannum -refchanvel -restfreq -molecule -transition

39 History of a data set Location +Instrument.locationName latitude -longitude -height Instrument -name -description -shortname -locationname -URL Provenance InstrumentConf Observation INSPIRED BY LAMB!&!POWER S RAW RADIO DM!NOTE AntennaConf Feed BeamConf Antenna -name -polarisation: L,R,X,Y -name +AntennaConf.name -mount -majoraxis -minoraxis -effectivearea RADAMS This set is defined for each instrument band AND receiver configuration { Beam +beamconf.name -beammajor -beamminor -sensitivity -meanbeamsolidangle -minorbeamsolidangle -directivity -gain -resolution Receiver +beamconf.name -skycentrefreq -ifcentrefreq -bandwidth Spectrum +beamconf.name -chanseparation -freqresol -velrefframe -refchannum -refchanfreq -restfreq -molecule -transition Velocity +beamconf.name -chanseparation -velresol -velrefframe -refchannum -refchanvel -restfreq -molecule -transition

40 History of a data set Observation Provenance AmbientConditions Processing InstrumentConf -timestamp ProcessingStep +Processing.timeStamp -kind -softwarepackage -parameter.name -parameter.kind -parameter.value Calibration +Processing.timeStamp -parameter.name -parameter.kind -parameter.value -parameter.sigma -parameter.calcoeff[n] RADAMS

41 History of a data set Observation Provenance InstrumentConf AmbientConditions -timestamp -opacity -temperature -humidity -wind OpacityCurve +AmbientConditions.timeStamp -opacity -skydipstart -azimuth -elevation.[n] -tsky.[n] -atmosphericmodel RADAMS

42 History of a data set Observation Provenance InstrumentConf TIME CHANGING AmbientConditions -timestamp -opacity -temperature -humidity -wind OpacityCurve +AmbientConditions.timeStamp -opacity -skydipstart -azimuth -elevation.[n] -tsky.[n] -atmosphericmodel RADAMS

43 History of a data set Provenance Observation Location +Instrument.locationName latitude -longitude -height Instrument -name -description -shortname -locationname -URL InstrumentConf AntennaConf Feed BeamConf Antenna -name -polarisation: L,R,X,Y -name +AntennaConf.name -mount -majoraxis -minoraxis -effectivearea This set is defined for each instrument band AND receiver configuration { Beam +beamconf.name -beammajor -beamminor -sensitivity -meanbeamsolidangle -minorbeamsolidangle -directivity -gain -resolution Receiver +beamconf.name -skycentrefreq -ifcentrefreq -bandwidth Spectrum +beamconf.name -chanseparation -freqresol -velrefframe -refchannum -refchanfreq -restfreq -molecule -transition Velocity +beamconf.name -chanseparation -velresol -velrefframe -refchannum -refchanvel -restfreq -molecule -transition filters grisms observatory gratings telescope instrument technique mode detector slits ambient objectives etc.

44 History of a data set Provenance Observation Location +Instrument.locationName latitude -longitude -height Instrument -name -description -shortname -locationname -URL Antenna InstrumentConf AntennaConf -name Feed -polarisation: L,R,X,Y -name BeamConf UNIFICATION OF PROVENANCE AT THE AMBIENT, TELESCOPE_CONF LEVELS: +AntennaConf.name -mount -majoraxis -minoraxis -effectivearea THE REST IS VERY HETEROGENOUS This set is defined for each instrument band AND receiver configuration { Beam +beamconf.name -beammajor -beamminor -sensitivity -meanbeamsolidangle -minorbeamsolidangle -directivity -gain -resolution Receiver +beamconf.name -skycentrefreq -ifcentrefreq -bandwidth Spectrum +beamconf.name -chanseparation -freqresol -velrefframe -refchannum -refchanfreq -restfreq -molecule -transition Velocity +beamconf.name -chanseparation -velresol -velrefframe -refchannum -refchanvel -restfreq -molecule -transition filters grisms observatory gratings telescope instrument technique mode detector slits ambient objectives etc.

45 History of a data set Can only be done with: Historical archive of: optical elements software (possibly with virtualization) configuration values / fudge factors ambient information DIMM seeing, opacity, conductivity

46 History of a data set Can only be done with: Historical archive of: optical elements software (possibly with virtualization) configuration values / fudge factors ambient information DIMM seeing, opacity, conductivity + SYSTEMATIC OBSERVING LOGS!

47 Level 0 Period Ownership Tracking Program Type RunID Level 3 Level 4 Program ID Project bibcode ProgID Level 1 Level 2 DataID Run ID Run ID Run ID Data Release ID Night Night Structured ReadMe Association IDs Night Log OB ID Reduced Science Data Master Calib. Data Ancillary Main File(s) Associated File(s) Raw File (DP_ID) File DP_ID DP_ID DPR CATG

48 Level 0 Period Ownership Tracking Program Type RunID Level 3 Level 4 Program ID Project bibcode ProgID Level 1 Level 2 DataID Run ID Run ID Run ID Data Release ID Night Night Structured ReadMe Association IDs Night Log OB ID Reduced Science Data Master Calib. Data Ancillary Main File(s) Associated File(s) Raw File (DP_ID) File DP_ID DP_ID DPR CATG

49 Ownership Tracking

50 Ownership Tracking Can only rely on unique identifiers being maintained Other VO manipulations (SAMP messages, crossmatching) can lose associations Need to provide services for identification of key IDs Increase of the role of the IVOA Registry? This is stopping some small publishers!

Ownership Tracking Can only rely on unique identifiers being maintained Other VO manipulations (SAMP messages, crossmatching) can lose associations Need to provide

51 Ownership Tracking Can only rely on unique identifiers being maintained Other VO manipulations (SAMP messages, crossmatching) can lose associations Need to provide services for identification of key IDs Increase of the role of the IVOA Registry? This is stopping some small publishers! THIS TRACKING CAN BE USED TO MEASURE PUBLICATION!RATIOS

52 Quality Assessment

53 Quality Assessment Linked to: Acquisition configuration Actual problems with instruments/telescope Weather Intended usage of the dataset(s)!

54 Quality Assessment Linked to: Acquisition configuration Actual problems with instruments/telescope Weather Intended usage of the dataset(s)! NO OBJECTIVE, ONE-SIZE-FITS-ALL, QUALITY ASSESSMENT METRIC

55 Quality Assessment Linked to: Acquisition configuration Actual problems with instruments/telescope Weather Intended usage of the dataset(s)!

56 Quality Assessment Linked to: Acquisition configuration TIME CHANGING Actual problems with instruments/telescope Weather Intended usage of the dataset(s)!

57 Queriable Provenance? Most queries, on Characterisation/Target/Curation Instrument-specific forms for instrument configurations Allow UType/UCD/IVOAT key-value pairs? Instrument-specific a priori quality assessment A posteriori, usage-specific quality assessment, offline > usage-specific query to data Provenance

58 Conclusions

59 Conclusions Provenance is an integral part of the Observation DM Provenance comes with discipline, but allows for quality science Different approach for different kinds of instruments, but all under the same general framework Provenance should be accessible for any item; specialised TAP version?

60 Conclusions Data centres: consistent naming/coding, plus mappings for existing data History, history, history! Should we forget about past data, and focus in the future?

61 Vielen Dank! STARTING WITH MY GERMAN CLASSES ;-)

Visualisation and analysis of 3D datasets in the Virtual Observatory

Collaborators: 3D2008 Workshop ESO Garching 12 Jun 2008 Visualisation and analysis of 3D datasets in the Virtual Observatory follow-up of the talks at ADASS-2005 and Spec&VO-2007 Igor Chilingarian LERMA