The SDSS, Databases and SQL Just another day in database land

Transcription

1 The SDSS, Databases and SQL Just another day in database land

2 Overview of the day A more in-depth discussion of how data bases are put together. Searching data - why databases are useful. The Structured Query Language (SQL) Examples of astronomical databases & other useful tools.

3 The SDSS - data & how to get it

4 What we will cover The Sloan Digital Sky Survey (SDSS) The general characteristics of the survey The data obtained and the quantities measured Gotcha s Navigation of the SDSS online data Databases - a slight detour SQL - clever searching Extracting data from the SDSS and making use of it all

5 (Some) Goals You should be able to navigate the SDSS web site and know the difference between the CAS and DAS. You should be able to find images of a particular area of sky & check which objects have been observed spectroscopically. You should know what to watch out for in the data. You should be able to download FITS images and spectra for the objects that interest you You should know how to write simple SQL queries and should know where to find information for more complex queries and when to use them.

6 What might it be useful for during your research work? You might be able to use the SDSS to create a comparison sample for your analysis. E.g. find similar stars, galaxy clusters and extract these to put your results in context. You might need to use data from the SDSS directly. Finding-charts and images of the sky. The SQL you learn might be useful if you need to interact with databases in the future.

7 The SDSS The most ambitious survey of the sky ever undertaken. Imaging survey of 8600 square degrees. Redshifts of more than 1,000,000 galaxies & QSOs. Robotic 2.5m telescope - imaging & Spectrscopy

8 The SDSS DR6 - (Summer 2007) 5 band (ugriz) imaging of 8520 deg 2 [~10Tb]. 3x10 8 unique objects. R~1800 spectroscopy covering Å for 1,271,680 objects [793,358 galaxies, 105,499 QSOs, 218,672 stars including 70,398 M and later]. Data kept in a searchable (SQL) data base. Main galaxy sample: r < also a QSO sample, Luminous Red Galaxies sample + stars & special objects. DR6 is the first release for SDSS-II: SEGUE & stellar parameters: [Fe/H], T eff, log g for all stars. Also Legacy & SDSS SNe. Median seeing for images: 1.43 (50%: [1.3, 1.58 ])

9 DR5

10 DR5

11 DR5 Sloan Great Wall

12 The Workings Images are taken in a drift-scan using a mosaic camera with five filters. Each scan is called a strip, but to cover gaps a second scan is made and the combination is called a stripe. The exposure time is ~54s with ~73s between each filter. The images are then analysed using a code called Photo and based on the measurements objects are selected for spectroscopy. Spectroscopy is carried out the following season using a fibre spectrograph with 3 fibres covering 3800Å-9000Å. Spectroscopic data are analysed using two pipelines and imaging & spectroscopy is released to the public.

13 Normal magnitudes: Magnitude System(s) m=-2.5 Log 10 f + zp asinh magnitudes (luptitudes): (f/f 0 ) 2 b m=-2.5/ln(10) [asinh( ) + ln b] The difference is small (irrelevant) for bright objects but is very important at faint flux levels! nano-maggies: A linear flux measurement such that 1 nmgy corresponds to a conventional magnitude of Widely used in the NYU Value Added Galaxy Catalogue (VAGC)

14 Filter System

15 Magnitude System(s) Relationship to other photometric calibrations: The SDSS is a near AB magnitude system. Thus conversion to Janskys is fairly easy: AB=2.5 ( 23 - log 10 F [Jy] ) But there are some small offsets, such that: u AB = u SDSS & z AB = z SDSS To convert to Johnson etc systems: An example (for stars, from Robert Lupton): B = g *(g - r) V = g *(g - r) R = r *(r - i) I = i *(i - z)

16 Magnitude System(s) Relationship to other photometric calibrations: The SDSS is a near AB magnitude system. Thus conversion to Janskys is fairly easy: Be aware that there now is AB=2.5 ( 23 - log 10 F [Jy] ) Ubercalibration: Reduced But there are some small offsets, such that: u AB = u SDSS & z AB = z SDSS systematic effects across the entire survey (~1%). To convert to Johnson etc systems: (Padmanabhan et al 2007) An example (for stars, from Robert Lupton): B = g *(g - r) V = g *(g - r) R = r *(r - i) I = i *(i - z)

17 The offerings: Photometry Photometric data comes from Photo:

18 The offerings: Photometry Photometric data comes from Photo: Magnitudes [ugriz, Petrosian, Model, Apertures, PSF]

19 The offerings: Photometry Photometric data comes from Photo: Magnitudes [ugriz, Petrosian, Model, Apertures, PSF] Sizes [Petrosian radii, Image moments]

20 The offerings: Photometry Photometric data comes from Photo: Magnitudes [ugriz, Petrosian, Model, Apertures, PSF] Sizes [Petrosian radii, Image moments] Positions [better than 0.1, often multiple observations]

21 The offerings: Photometry Photometric data comes from Photo: Magnitudes [ugriz, Petrosian, Model, Apertures, PSF] Sizes [Petrosian radii, Image moments] Positions [better than 0.1, often multiple observations] Shapes, orientations [image moments]

22 The offerings: Photometry Photometric data comes from Photo: Magnitudes [ugriz, Petrosian, Model, Apertures, PSF] Sizes [Petrosian radii, Image moments] Positions [better than 0.1, often multiple observations] Shapes, orientations [image moments] Images [full field, JPGs, objects only - atlas images ]

23 The offerings: Photometry Photometric data comes from Photo: Magnitudes [ugriz, Petrosian, Model, Apertures, PSF] Sizes [Petrosian radii, Image moments] Positions [better than 0.1, often multiple observations] Shapes, orientations [image moments] Images [full field, JPGs, objects only - atlas images ] Image quality [position dependent PSF as PCAs]

24 Reliability: The offerings: Photometry - Advice

25 The offerings: Photometry - Advice Reliability: The r-band is in general the most reliable - u & z might be poor quality.

26 The offerings: Photometry - Advice Reliability: The r-band is in general the most reliable - u & z might be poor quality. r<21.5 should in general be fine, but be aware of single-band detections (typical detection limit r AB ~ 22.5).

27 The offerings: Photometry - Advice Reliability: The r-band is in general the most reliable - u & z might be poor quality. r<21.5 should in general be fine, but be aware of single-band detections (typical detection limit r AB ~ 22.5). r<14.5 requires care and sky estimates for very large galaxies are often poor. See e.g. Blanton et al (2005) for details.

28 The offerings: Photometry - Advice Reliability: The r-band is in general the most reliable - u & z might be poor quality. r<21.5 should in general be fine, but be aware of single-band detections (typical detection limit r AB ~ 22.5). r<14.5 requires care and sky estimates for very large galaxies are often poor. See e.g. Blanton et al (2005) for details. Large, actively star forming galaxies are sometimes shredded with multiple spectroscopic targets.

29 The offerings: Photometry - Advice Reliability: The r-band is in general the most reliable - u & z might be poor quality. r<21.5 should in general be fine, but be aware of single-band detections (typical detection limit r AB ~ 22.5). r<14.5 requires care and sky estimates for very large galaxies are often poor. See e.g. Blanton et al (2005) for details. Large, actively star forming galaxies are sometimes shredded with multiple spectroscopic targets. Take care to check imaging flags.

30 The offerings: Photometry - Advice Reliability: The r-band is in general the most reliable - u & z might be poor quality. r<21.5 should in general be fine, but be aware of single-band detections (typical detection limit r AB ~ 22.5). r<14.5 requires care and sky estimates for very large galaxies are often poor. See e.g. Blanton et al (2005) for details. Large, actively star forming galaxies are sometimes shredded with multiple spectroscopic targets. Take care to check imaging flags. What should I use? For colours: Model magnitudes or aperture magnitudes. For total flux: Petrosian or optimal model magnitudes. Fiber magnitudes are useful to check spectrophotometry.

31 The offerings: Photometry - Advice Reliability: The r-band is in general the most reliable - u & z might be poor quality. r<21.5 should in general be fine, but be aware of single-band detections (typical detection limit r AB ~ 22.5). r<14.5 requires care and sky estimates for very large galaxies are often poor. See e.g. Blanton et al (2005) for details. Large, actively star forming galaxies are sometimes shredded with multiple spectroscopic targets. Take care to check imaging flags. What should I use? For colours: Model magnitudes or aperture magnitudes. For total flux: Petrosian or optimal model magnitudes. Fiber magnitudes are useful to check spectrophotometry.

32 Imaging Flags... These are set as individual bits in a long integer to indicate whether all was well with the photometric analysis. It is crucial to check these when doing accurate work! Some examples: SATURATED: Tells you whether the image of an object was saturated. EDGE: Is the object on the edge (commonly the case for large galaxies). CHILD: Is this object part of a larger object that was split? MOVED: Did the object move? etc. etc. See flags.html for details.

33 The offerings: Spectroscopy Spectroscopic data comes from spectro1d:

34 The offerings: Spectroscopy Spectroscopic data comes from spectro1d: Absorption line indices [Lick + a few more]

35 The offerings: Spectroscopy Spectroscopic data comes from spectro1d: Absorption line indices [Lick + a few more] Emission lines [Fluxes, EWs, widths]

36 The offerings: Spectroscopy Spectroscopic data comes from spectro1d: Absorption line indices [Lick + a few more] Emission lines [Fluxes, EWs, widths] Velocity dispersions [Two pipelines, Elodie PCA]

37 The offerings: Spectroscopy Spectroscopic data comes from spectro1d: Absorption line indices [Lick + a few more] Emission lines [Fluxes, EWs, widths] Velocity dispersions [Two pipelines, Elodie PCA] Probably the area where using Value-Added Catalogues is a useful alternative! In particular the MPA-JHU database with more careful continuum subtraction and the NYU VAGC which has more matching and is an excellent base sample for further work.

38 Reliability: The offerings: Spectroscopy - Advice

39 Reliability: The offerings: Spectroscopy - Advice [O II]3727Å is not measured for low-z galaxies.

40 Reliability: The offerings: Spectroscopy - Advice [O II]3727Å is not measured for low-z galaxies. Very strong emission lines are sometimes clipped.

41 Reliability: The offerings: Spectroscopy - Advice [O II]3727Å is not measured for low-z galaxies. Very strong emission lines are sometimes clipped. Bright sky and/or CCD defects might cause some lines not to be measured.

42 Reliability: The offerings: Spectroscopy - Advice [O II]3727Å is not measured for low-z galaxies. Very strong emission lines are sometimes clipped. Bright sky and/or CCD defects might cause some lines not to be measured. Sky subtraction is not very good in the red [c.f. Wild et al 2005]

43 Reliability: The offerings: Spectroscopy - Advice [O II]3727Å is not measured for low-z galaxies. Very strong emission lines are sometimes clipped. Bright sky and/or CCD defects might cause some lines not to be measured. Sky subtraction is not very good in the red [c.f. Wild et al 2005] The spectrophotometric calibration for DR6 is tied to PSF magnitudes - watch out for galaxies!

44 Reliability: The offerings: Spectroscopy - Advice [O II]3727Å is not measured for low-z galaxies. Very strong emission lines are sometimes clipped. Bright sky and/or CCD defects might cause some lines not to be measured. Sky subtraction is not very good in the red [c.f. Wild et al 2005] The spectrophotometric calibration for DR6 is tied to PSF magnitudes - watch out for galaxies! The slope of the spectrum in the blue is only good to the ~few % level.

45 Reliability: The offerings: Spectroscopy - Advice [O II]3727Å is not measured for low-z galaxies. Very strong emission lines are sometimes clipped. Bright sky and/or CCD defects might cause some lines not to be measured. Sky subtraction is not very good in the red [c.f. Wild et al 2005] The spectrophotometric calibration for DR6 is tied to PSF magnitudes - watch out for galaxies! The slope of the spectrum in the blue is only good to the ~few % level.

46 Stellar analysis from SEGUE The offerings: SEGUE

47 Stellar analysis from SEGUE The offerings: SEGUE Absorption line indices [Lick + a few more]

48 Stellar analysis from SEGUE The offerings: SEGUE Absorption line indices [Lick + a few more] Stellar parameters [Teff, log g, [Fe/H], R V ]

49 Stellar analysis from SEGUE The offerings: SEGUE Absorption line indices [Lick + a few more] Stellar parameters [Teff, log g, [Fe/H], R V ] The stellar parameters are derived from a number of different pipelines. Since the spectra in the SDSS are not optimal to derive stellar parameters it is necessary to compare these estimates to assess the quality of the results! And also to get external data with better resolution. ([Fe/H] ~ 0 show noticeable systematic offset). SEGUE in general targets lower Galactic latitudes and crowded fields and go outside the normal SDSS footprint. Thus much of the data is not in the normal CAS context!

50 The offerings: Some Others The SDSS Supernova search - repeat observations of Stripe 82. These are sometimes taken in poor conditions and are therefore provided in uncalibrated form.

51 The offerings: Some Others The SDSS Supernova search - repeat observations of Stripe 82. These are sometimes taken in poor conditions and are therefore provided in uncalibrated form. Moving objects [from repeat observations]

52 The offerings: Some Others The SDSS Supernova search - repeat observations of Stripe 82. These are sometimes taken in poor conditions and are therefore provided in uncalibrated form. Moving objects [from repeat observations] Variability [from repeat observations]

53 The offerings: Some Others The SDSS Supernova search - repeat observations of Stripe 82. These are sometimes taken in poor conditions and are therefore provided in uncalibrated form. Moving objects [from repeat observations] Variability [from repeat observations] Observing conditions [extinction, seeing etc.]

54 The offerings: Some Others The SDSS Supernova search - repeat observations of Stripe 82. These are sometimes taken in poor conditions and are therefore provided in uncalibrated form. Moving objects [from repeat observations] Variability [from repeat observations] Observing conditions [extinction, seeing etc.] Raw data [raw counts etc.]

55 Value-Added Data Very important! When people have done some careful job they might offer this as a value-added catalogue. Using these, rather than the official SDSS data might save you a lot of time and effort! Finding out about these: Read papers!! Check the SDSS VAC page: Listen carefully - as I will say this only once

56 Value-Added Data The NYU VAGC Large-Scale Structure samples with well characterised selection functions. Cross-matches of SDSS to other surveys. MPA-JHU VAGC Improved spectroscopic data reduction, line fluxes and indices. Derived quantities such as SFR, O/H and stellar masses etc. QSO catalogues (Schneider et al) & BAL-QSOs. White Dwarf catalogues (Eisenstein et al) Variable star catalogues from Stripe 82 & CV catalogues Moving objects catalogues Galaxy cluster catalogues (MaxBCG, cut & enhance) & more!

57 See: Blanton et al (2005) The NYU VAGC Large-scale structure sample. Careful characterisation of the survey geometry. Low-z galaxy sample (this is non-trivial to construct). SDSS data sweeps - files with a useful subset of all the data with manageable size (~100 Gb). Sersic fits & a number of other quantities. Various pieces of software - in particular kcorrect which is the most widely used software to calculate k- corrections for SDSS data.

58 The MPA-JHU VAGC See e.g: Tremonti et al (2004); Brinchmann et al (2004); Improved continuum subtraction. Pipeline optimised to measure emission lines and absorption line indices on galaxy spectra. Emission lines, absorption line indices. Star formation rates, oxygen abundances, stellar masses, emission line classification of galaxies and stellar metallicities. All the photometric information available in tsobj files for spectroscopic targets. Data organised in a set of FITS files and presently only for DR4 but DR7 is on the way.

59 Navigating it all

60 Navigating it all

61 The Web Interfaces The web site: Interactive browsing of the sky, finding charts, generic entry point etc. The DAS (Data Archive Server) Access to individual files - flatfiles. For large sets of files and for automatic download The CAS (Catalog Archive Server) Access to the database interfaces Google Earth & Sky

62 Navigate Point and click interface to browsing the sky. Intuitive and with useful overlays of spectroscopic/photometric targets etc. Inspection of individual objects provides much more information and direct access to flat files and NED/Simbad etc. Extremely useful for checking individual objects, problems & spectroscopic targeting.

63 Image Lists Allows the submission of a list of objects to get access to their images and provides links to navigate further. Very useful when matching to old catalogues with moderately good astrometry for instance.

64 Image Lists Allows the submission of a list of objects to get access to their images and provides links to navigate further. Very useful when matching to old catalogues with moderately good astrometry for instance.

65 Databases In astronomy and real life. Searching and using

66 The need for Databases Very large data sets require advanced techniques for analysis. Most researchers lack the skills to do e.g. correlation functions for >10 8 objects. Normal files (e.g. text files, FITS files) are inconvenient when >1000 properties are measured for each object. The indexing properties of databases make searches in general very fast and flexible and allow for powerful combinations of tables. It is even useful outside of astronomy... :)

67 A practical example Let us take a specific example, first a non-astronomy one: A photo sharing web-site. 1. What is the info we want to keep? 2. How do we want to organise it? What do you want to do?

68 A practical example Let us take a specific example, first a non-astronomy one: A photo sharing web-site. 1. What is the info we want to keep? 2. How do we want to organise it? What do you want to do? Photographer Location Image data Camera

69 A practical example Let us take a specific example, first a non-astronomy one: A photo sharing web-site. 1. What is the info we want to keep? 2. How do we want to organise it? What do you want to do? Photographer Location Image data ID: 1 Name: JB Loc: Data: <...> Camera: Nikon Camera

70 A practical example Let us take a specific example, first a non-astronomy one: A photo sharing web-site. 1. What is the info we want to keep? 2. How do we want to organise it? What do you want to do? Photographer Location Image data Camera ID: ID: 1 ID: 1 Name: ID: 1 Name: ID: 1JB Name: ID: 1JB Loc: Name: Loc: ID: 1JB Name: Loc: ID: 12 JB Name: JB Data: Loc: Name: JB Data: Loc: Name: <...> 03-04JB Data: <...> 03-04JB Loc: <...> Camera: Data: Loc: <...> Camera: Data: Loc: Nikon <...> Camera: Data: Nikon <...> Camera: Data: Nikon <...> Camera: Data: Nikon <...> Camera: Nikon Camera: Nikon Camera: Nikon Canon

71 A practical example Let us take a specific example, first a non-astronomy one: A photo sharing web-site. 1. What is the info we want to keep? 2. How do we want to organise it? What do you want to do? Photographer Location Image data Camera ID: ID: 1 ID: 1 Name: ID: 1 Name: ID: 1JB Name: ID: 1JB Loc: Name: Loc: ID: 1JB Name: Loc: ID: 12 JB Name: JB Data: Loc: Name: JB Data: Loc: Name: <...> 03-04JB Data: <...> 03-04JB Loc: <...> Camera: Data: Loc: <...> Camera: Data: Loc: Nikon <...> Camera: Data: Nikon <...> Camera: Data: Nikon <...> Camera: Data: Nikon <...> Camera: Nikon Camera: Nikon Camera: Nikon Canon Table of photos

72 A practical example A photo sharing web-site. 2. How do we want to organise it? What do you want to do? Would like to find all photos from a given photographer Get me all photos taken with a Nikon Find all photos close to a given location Find all photos with similar colours to a selected photo etc. etc. Need more than one table?

73 Table of photos ID: 2 Name: ID: 3 JB Loc: Name: 05-04JB Data: Loc: <...> Camera: Data: <...> Canon Camera: Canon Having multiple tables can save a lot of space but there must be something that links them - these are often index quantities. Table of locations Table of cameras This would be a version of a relational database (although it is unimportant what that really is)

74 The astronomical case Object info: MyGal SBc IMAGE Obs info: May :32:01-00:00: CLEAR K S 60S OK Table of Table of objects Observations Table of Galaxies Table of Stars

75 A simple version of the SDSS model Object ID Image Position u,g,r,i,z Profiles Object ID Object ID SpecObjID Spectra Redshift Emission line flux SpecObj

76 A simple version of the SDSS model Object ID Image SpecObjID Spectra Position u,g,r,i,z Profiles Redshift Emission line flux SpecObj Object ID Object ID Note that a given position can pay host to different objects! Note also that the photometric objects really live in a separate data base table from the spectroscopic objects - the keys connect the two. Note also that there is a lot of information outside this structure e.g.:

77 A simple version of the SDSS model Object ID Image SpecObjID Spectra Position u,g,r,i,z Profiles Redshift Emission line flux SpecObj Object ID Object ID Note that a given position can pay host to different objects! Note also that the photometric objects really live in a separate data base table from the spectroscopic objects - the keys connect the two. Note also that there is a lot of information outside this structure e.g.: X-ray data, common names ++

78 Interplay Web server HTML pages Browser User (typically) XML Data base Often SQL User interface server Efficient data transfer Data store (local or anywhere on the net)

79 The SDSS Database Tables PhotoObjAll, SpecObjAll, Photoz, RC3, etc... These contain the different sets of data and form the backbone Views PhotoObj, SpecObj, Galaxy, Star, Sky etc... These are provide convenient interfaces to Tables. This is often what you want to use! Functions fdistanceeq, ffield, fgetnearbyobjeq etc... Functions that might be very useful in queries and can also save you quite a bit of work. They are particularly powerful when combined with iteration To find out exactly what is available: Look at the Schema

80 The SDSS Database Tables PhotoObjAll, SpecObjAll, Photoz, RC3, etc... These contain the different sets of data and form the backbone Important: Views Best vs. Target PhotoObj, SpecObj, Galaxy, Star, Sky etc... These are provide convenient interfaces to Tables. This is often what you want to use! The photometry that was used when Functions fdistanceeq, ffield, fgetnearbyobjeq etc... Functions deciding that might spectroscopic be very useful in targets queries and is stored can also save you in Target quite a bit - this of work. is often They are different particularly from powerful when combined with iteration the currently Best reductions. You have to choose one of these two contexts. To find out exactly what is available: Look at the Schema

81 The SDSS Database Tables Important: PhotoObjAll, SpecObjAll, Photoz, RC3, etc... These contain the different sets of data and form the backbone Primary & SciencePrimary Important: Views Best vs. Target PhotoObj, SpecObj, Galaxy, Star, Sky etc... Each real object on the sky should have one Primary These are provide convenient interfaces to Tables. This is often what you want to use! object associated with it. They might also have The photometry that was used when Secondary objects associated but you will often only Functions fdistanceeq, ffield, fgetnearbyobjeq etc... need Functions deciding primary that might objects. spectroscopic be very useful in targets queries and is stored can also save you in Target quite a bit - this of work. is often They are different particularly from powerful when combined with iteration For spectra there is a similar sounding quantity called the currently Best reductions. You have SciencePrimary - this indicates whether the spectrum to choose one of these two contexts. To find out exactly what is available: is considered good enough quality AND that this is not a re-observation. Look at the Schema

82 Look for keys Access to various parts of the database The view, table or function

83 Databases & SQL

84 Structured Query Language - SQL A computer language designed for efficient query of databases. It is used both to create tables/databases and to search these. In our work we are often only concerned with the search aspect: SELECT specifies what is to be returned (this is the command) FROM specifies what table to use WHERE is an optional clause that specifies a subset of the full data. But be aware that for advanced searches it is often necessary/convenient to be able to CREATE temporary tables. (we will return to this) Combinations of tables can be done using JOINs. These require that there are entries in each table that are in common.

85 Structured Query Language - SQL SELECT objid, u, g, r, FROM PhotoPrimary WHERE u - g < 0.4 and g - r < 0.7

86 Structured Query Language - SQL SELECT objid, u, g, r, Select something FROM PhotoPrimary WHERE u - g < 0.4 and g - r < 0.7

87 Structured Query Language - SQL SELECT objid, u, g, r, Select something FROM PhotoPrimary WHERE u - g < 0.4 and g - r < 0.7 From a table/view

88 Structured Query Language - SQL SELECT objid, u, g, r, Select something FROM PhotoPrimary WHERE u - g < 0.4 and g - r < 0.7 From a table/view According to a criterion

89 Some examples: Count galaxies: SELECT COUNT(*) AS "Low redshift galaxies" FROM SpecObj WHERE z BETWEEN AND 0.2

90 Some examples: Count galaxies: SELECT COUNT(*) AS "Low redshift galaxies" FROM SpecObj WHERE z BETWEEN AND 0.2 SELECT COUNT(*) AS "Low redshift galaxies"

91 Some examples: Count galaxies: SELECT COUNT(*) AS "Low redshift galaxies" FROM SpecObj WHERE z BETWEEN AND 0.2 SELECT COUNT(*) AS "Low redshift galaxies" SQL Statement: we want the following returned

92 Some examples: Count galaxies: SELECT COUNT(*) AS "Low redshift galaxies" FROM SpecObj WHERE z BETWEEN AND 0.2 SELECT COUNT(*) AS "Low redshift galaxies" SQL Statement: we want the following returned Count all matches to the WHERE statement below.

93 Some examples: Count galaxies: SELECT COUNT(*) AS "Low redshift galaxies" FROM SpecObj WHERE z BETWEEN AND 0.2 SELECT COUNT(*) AS "Low redshift galaxies" SQL Statement: we want the following returned Count all matches to the WHERE statement below. Alias for the result - in some systems the return data would be named according to this.

94 Some examples: Count galaxies: SELECT COUNT(*) AS "Low redshift galaxies" FROM SpecObj WHERE z BETWEEN AND 0.2

95 Some examples: Count galaxies: SELECT COUNT(*) AS "Low redshift galaxies" FROM SpecObj WHERE z BETWEEN AND 0.2 FROM SpecObj

96 Some examples: Count galaxies: SELECT COUNT(*) AS "Low redshift galaxies" FROM SpecObj WHERE z BETWEEN AND 0.2 FROM SpecObj SQL Statement: use the following table or view.

97 Some examples: Count galaxies: SELECT COUNT(*) AS "Low redshift galaxies" FROM SpecObj WHERE z BETWEEN AND 0.2 FROM SpecObj SQL Statement: use the following table or view. The table to carry out the search within.

98 Some examples: Count galaxies: SELECT COUNT(*) AS "Low redshift galaxies" FROM SpecObj WHERE z BETWEEN AND 0.2

99 Some examples: Count galaxies: SELECT COUNT(*) AS "Low redshift galaxies" FROM SpecObj WHERE z BETWEEN AND 0.2 WHERE z BETWEEN AND 0.2

100 Some examples: Count galaxies: SELECT COUNT(*) AS "Low redshift galaxies" FROM SpecObj WHERE z BETWEEN AND 0.2 WHERE z BETWEEN AND 0.2 SQL Statement: Only include the objects satisfying the following constraints

101 Some examples: Count galaxies: SELECT COUNT(*) AS "Low redshift galaxies" FROM SpecObj WHERE z BETWEEN AND 0.2 WHERE z BETWEEN AND 0.2 SQL Statement: Only include the objects satisfying the following constraints Only objects with redshifts between and 0.2

102 A small worked example Scientific Question: How does the metal content of stars vary as a function of Galactic latitude? Step 1: Decide what data you require. We need positions & metallicity. Ra, Dec & [Fe/H] (+ quality, flags etc) Step 2: Determine what tables you can get this from. Do you need VACs? sppparams contains what we want (and more) Step 3: Write an appropriate SQL statement & download data Step 4: Read in data & carry out post-processing to visualize the results.

103 A small example Question: How does the metal content of stars vary as a function of Galactic latitude? Step 3: Write an appropriate SQL statement & download data SELECT fehw, ra, dec FROM sppparams WHERE fehwn > 3

104 A small example Question: How does the metal content of stars vary as a function of Galactic latitude? Step 3: Write an appropriate SQL statement & download data SELECT fehw, ra, dec This is a weighted average of [Fe/H] estimates. FROM sppparams WHERE fehwn > 3

105 A small example Question: How does the metal content of stars vary as a function of Galactic latitude? Step 3: Write an appropriate SQL statement & download data SELECT fehw, ra, dec FROM sppparams WHERE fehwn > 3 This is a weighted average of [Fe/H] estimates. This ensures that only stars with at least 4 different estimates of [Fe/H] are chosen. (In practice I did this within CASJobs and then published the resulting catalogue and downloaded it as a FITS file)

106 A small example Question: How does the metal content of stars vary as a function of Galactic latitude? Step 4: Read in data & carry out post-processing to visualize the results. Convert Ra, Dec to l, b Plot [Fe/H] as a function of l, b and b alone.

107 A small example Question: How does the metal content of stars vary as a function of Galactic latitude? Step 4: Read in data & carry out post-processing to visualize the results. Convert Ra, Dec to l, b Plot [Fe/H] as a function of l, b and b alone.

108 Photometric table ID 1 Spectroscopic table u 19.3 ID 1 ID 2 EW(Ha) 75Å u 17.5 ID 3 ID 3 EW(Ha) 0.5Å u 20.5

109 Photometric table How do you combine these? ID 1 Spectroscopic table u 19.3 ID 1 ID 2 EW(Ha) 75Å u 17.5 ID 3 ID 3 EW(Ha) 0.5Å u 20.5

110 First a theoretical view: Two sets of values: {x i } {y j } (the elements can be vectors/matrices etc) Possible ways to combine:

111 First a theoretical view: How do you combine these? Two sets of values: {x i } {y j } (the elements can be vectors/matrices etc) Possible ways to combine:

112 First a theoretical view: How do you combine these? Two sets of values: {x i } {y j } (the elements can be vectors/matrices etc) Possible ways to combine: Union: {x i, y j i=1, n; j=1, m} elements must be the same

113 First a theoretical view: How do you combine these? Two sets of values: {x i } {y j } (the elements can be vectors/matrices etc) Possible ways to combine: Union: {x i, y j i=1, n; j=1, m} elements must be the same Cross-join: {(x i, y j ) i=1, n; j=1, m} ie. all possible pairs

114 First a theoretical view: How do you combine these? Two sets of values: {x i } {y j } (the elements can be vectors/matrices etc) Possible ways to combine: Union: {x i, y j i=1, n; j=1, m} elements must be the same Cross-join: {(x i, y j ) i=1, n; j=1, m} ie. all possible pairs Outer join: {(x i, y i ) if y i exists, (x i, NULL) otherwise}

115 First a theoretical view: How do you combine these? Two sets of values: {x i } {y j } (the elements can be vectors/matrices etc) Possible ways to combine: Union: {x i, y j i=1, n; j=1, m} elements must be the same Cross-join: {(x i, y j ) i=1, n; j=1, m} ie. all possible pairs Outer join: {(x i, y i ) if y i exists, (x i, NULL) otherwise} Inner join: {(x i, y i ) if y i exists}

116 How do you combine these? ID 1 u 19.3 ID 1 ID 2 EW(Ha) 75Å u 17.5 ID 3 ID 3 EW(Ha) 0.5Å u 20.5

117 How do you combine these? ID u ID 1 JOIN ID u EW(Ha) ID u EW(Ha) ID 75Å Å ID u EW(Ha) 0.5Å Å

118 How do you combine these? ID u ID 1 JOIN ID u EW(Ha) ID u EW(Ha) ID 75Å Å ID u EW(Ha) 0.5Å Å SELECT P.u, S.z FROM Photo as P JOIN Spectro as S ON P.ID=S.ID

119 How do you combine these? ID 1 u 19.3 ID 1 ID 2 EW(Ha) 75Å u 17.5 ID 3 ID 3 EW(Ha) 0.5Å u 20.5

120 How do you combine these? ID u ID 1 JOIN ID u EW(Ha) ID u EW(Ha) ID 75Å Å ID u EW(Ha) 0.5Å Å

121 How do you combine these? ID u ID 1 JOIN ID u EW(Ha) ID u EW(Ha) ID 75Å Å ID u EW(Ha) 0.5Å Å But if we want to keep all possible pairs we need an OUTER JOIN

122 How do you combine these? ID u ID 1 JOIN ID u EW(Ha) ID u EW(Ha) ID 75Å Å ID u EW(Ha) 0.5Å Å But if we want to keep all possible pairs we need an OUTER JOIN SELECT P.u, S.z FROM Photo as P LEFT OUTER JOIN Spectro as S ON P.ID=S.ID

123 How do you combine these? ID u ID 1 Join ID u EW(Ha) ID u EW(Ha) ID 75Å Å ID u EW(Ha) 0.5Å Å But if we want to keep all possible pairs we need an OUTER JOIN SELECT P.u, S.z FROM Photo as P LEFT OUTER JOIN Spectro as S ON P.ID=S.ID ID u EW(Ha) Å NULL Å

124 Joins - Combining Tables Table A (e.g. PhotoAll) ID1 u g... Table B (e.g. SpecObjAll) ID1 ID2 Hα z Normal language: Get z from Table B and g from Table A for the same object. SQL: Need a quantity that is common between two tables. Here ID1. Then create a join - where A

125 Joins - Combining Tables Table A (e.g. PhotoAll) ID1 u g... Table B (e.g. SpecObjAll) ID1 ID2 Hα z SELECT A.u, A.g, B.z FROM PhotoAll as A, SpecObjAll as B WHERE A.ID1=B.ID1 Notice that we here use the as construct to give tables a nickname.

126 Joins - Combining Tables Table A (e.g. PhotoAll) ID1 u g... Table B (e.g. SpecObjAll) ID1 ID2 Hα z SELECT A.u, A.g, B.z FROM PhotoAll as A LEFT OUTER JOIN SpecObjAll as B ON A.ID1=B.ID1 The most common left outer join in SDSS is between photometry and spectroscopy - for this reason there is a view called SpecPhoto which contains this join pre-computed. Use this when you want both spectroscopic info and photometry.

127 Looking at an SDSS Join Find galaxies with EW(Ha) > 40Å SELECT G.ObjID -- we want the photometric ObjID FROM Galaxy as G, SpecObj as S, SpecLine as L WHERE G.ObjID = S.bestObjID -- the spectroscopic object should -- be (photometrically) a galaxy and S.SpecObjID = L.SpecObjID -- and spectral line L is detected in spectrum and L.LineId = 6565 and L.ew > and line L is the H alpha line -- and H alpha is at least angstroms wide

128 Looking at an SDSS Outer Join Find stars with & without spectra in a range: SELECT TOP 20 S.ra, S.dec, S.objID, S.specObjID FROM Star as S LEFT OUTER JOIN SpecObj as Sp ON S.ObjID = Sp.bestObjID WHERE S.ra > 180 AND S.ra < 181 AND abs(s.dec) < 1

129 Ordering & Groupings If your search would return, say 100 million objects and you only want to count them in some bins, you might not want to download it all. The solution? GROUP BY. Disordered/randomly ordered lists annoying you? ORDER BY is your friend!

130 Ordering & Groupings An example: SELECT.2*(.5+floor(g.r/.2)) as mag, count(*) as num FROM GALAXY group by.2*(.5+floor(g.r/.2)) order by mag

131 Finding Stuff How do you find stuff quickly? Look-up tables/indices, clever organisation Unique IDs (all members of database have this) Sorting the data Maybe duplicate data to allow for several sortings

132 But what about in space? Table A Table B For N & M objects we would naively need to calculate NxM distances - time-consuming for large databases. So what can be done?

133 But what about in space? Table A Table B For N & M objects we would naively need to calculate NxM distances - time-consuming for large databases. So what can be done? A grid is one possibility (used in the Millennium database for instance). Naming becomes important so that closeness can be defined. Well-suited to Euclidean spaces

134 But what about in space? Table A Table B For N & M objects we would naively need to calculate NxM distances - time-consuming for large databases. So what can be done? Triangles is another possibility - this is used by the SDSS & quite a few other databases in what is called Hierarchical Triangular Mesh (HTM)

135 S1 The sphere can be divided into 8 triangular regions. By starting on the midpoint of each side of a triangle you can subdivide it and create new triangles & then continue. Triangles allow for a compact & (relatively) easy to understand naming scheme. The name is known as the HTMId. Since subdivisions keep the name of their parents it is easy to find regions that are close to a given one.

136 Do I need to know? It is useful as a technique in many programming situations. Finding objects that are close together in one way or another (don t just think about x, y, z or ra & dec!) is very often important. Usually you can use database functions to do the dirty work but you need to know what not to do. But, you will rarely need to know the details.

137 Other ways of getting data

138 Flat-Files The SDSS data are kept in a set of FITS files stored at Fermilab - in the DAS. This the reference set of data and is a superset of the CAS. Advantages: Can be analysed off-line using your own code All data available You will be able to measure quantities that have not been measured by the SDSS pipelines. Provided in a manner that matches the observing strategy Disadvantages: Contains often a lot more than what you want Requires some understanding of the survey observing strategy Requires a lot of disk space to store in full - searching the data subsequently requires significant effort.

139 When do you need them? Flat-Files You want to analyse everything You need/prefer to work off-line SQL makes the hair on your back stand up You need data that is not in the database (e.g. images, spectra) You have a specialized analysis pipeline you prefer to use. e.g: Spectroscopic analysis pipeline with better continuum subtraction (Tremonti et al 2004; Brinchmann et al 2004) Improved/specialised image analysis - improved treatment of large galaxies for instance.

140 Flat-Files How do you get them? The Data Archive Server (DAS) You can either browse the file system or create a file containing the necessary information to identify your required files and then upload this to the DAS. Some time later you will be told where to get your data.

141 Flat-Files How do you get them? The Data Archive Server (DAS) What can you get? fpc - calibrated image frames tsobj - One file per fpc with all detected objects spplate - 2D spectra for one plate with 640 fibres spspec - 1D spectra for a plate with measurements psfield - Extinction, PSFs etc. +++ Many more +++

142 Going Further

143 VO access Many key players in SDSS are involved in the Virtual Observatory so this can be used to access the data. Direct access to images in Aladin. Sky search services like OpenSkyQuery ( Spectrum services (Hungary, Baltimore) A number of web services are currently available. The JHU developer website is a good start: skyservice.pha.jhu.edu/develop/vo/

144 Some Hints CASJobs allows the creation of your own tables. This allows complex queries to be made. You need to know how to create tables. You might also need to use Procedures. You can then create a table and loop over this to put final results in a CASJobs table. Google Sky allows you to create KML files to overlay your own information on the sky. Can be useful for matching. And you can write your own scripts to interact with the SDSS - personally I often run these via the casjobs.jar Java library provided by JHU.

145 Other databases

146 The Millennium Simulation A very big simulation of the Universe hosted at the MPA in Garching. On top of this simulation a number of semianalytic models have been run and this has led to the construction of a number of simulated galaxy catalogues. These catalogues as well as halo catalogues are available through a SQL database with some example queries. Useful to know: A given redshift is known as a Snapnum The publicly available database is known as the millimillennium and contains 1/512 of the full simulation.

147 Examples of Use Calculating the Tully-Fisher relation (Luminosity versus Rotation velocity): SELECT vvir, mag_b, mag_k FROM millimil..delucia2006a WHERE (bulgemass < 0.1*stellarMass OR AND bulgemass IS NULL) snapnum = 41

148 Examples of Use Calculating the Tully-Fisher relation (Luminosity versus Rotation velocity): Velocity and magnitudes SELECT vvir, mag_b, mag_k FROM millimil..delucia2006a WHERE (bulgemass < 0.1*stellarMass OR bulgemass IS NULL) AND snapnum = 41

149 Examples of Use Calculating the Tully-Fisher relation (Luminosity versus Rotation velocity): SELECT vvir, mag_b, mag_k Table to use FROM millimil..delucia2006a WHERE (bulgemass < 0.1*stellarMass OR bulgemass IS NULL) AND snapnum = 41

150 Null values - SQLs dirty secret How to indicate that something is missing is actually not that easy - this means that the handling of null values in SQL varies quite a bit. It is usually best to use IS NULL to test explicitly for null values. Not doing that might or might not give you the results you want...

151 Examples of Use Calculating the Tully-Fisher relation (Luminosity versus Rotation velocity): SELECT vvir, mag_b, mag_k FROM millimil..delucia2006a WHERE (bulgemass < 0.1*stellarMass OR AND bulgemass IS NULL) snapnum = 41 What would happen if we left this out?

152 Examples of Use Calculating the Tully-Fisher relation (Luminosity versus Rotation velocity): SELECT vvir, mag_b, mag_k FROM millimil..delucia2006a WHERE (bulgemass < 0.1*stellarMass OR AND bulgemass IS NULL) snapnum = 41 What would happen if we left this out? We would miss out on all the galaxies without a bulge - late-type spirals!