Engineering considerations for large astrophysics projects

Size: px

Start display at page:

Download "Engineering considerations for large astrophysics projects"

Leon Harper
5 years ago
Views:

1 Engineering considerations for large astrophysics projects David W. Hogg Center for Cosmology and Particle Physics Department of Physics New York University Max-Planck-Institut für Astronomie Heidelberg, Germany 2014 January 9

2 punchlines Calibration programs are wasteful and reduce the accuracy of your end-of-mission results. (you will need to adjust your observing strategy) Homogeneity and uniformity of survey samples are impossible, unnecessary, and harmful goals. (you will need to implement some probability theory) Proper uncertainty propagation is not easy. (I got nothing) The challenge is to make precise measurements and keep discovery space open. (you will need to understand, quantitatively, your goals)

3 my teachers (incomplete list) Gerry Neugebauer (Caltech, emeritus) Sam Roweis (Toronto & NYU, deceased) Dave Schlegel (LBL) & Scott Burles (Cutler) Mike Blanton (NYU) Dustin Lang (CMU) & Jo Bovy (IAS) & Dan Foreman-Mackey (NYU)

4 survey-centric context Gaia SKA and pathfinders Euclid LSST SDSS-IV... and many more (I am going to get mean at the end.)

5 my day job Astrometry.net and TheTractor emcee and kplr precision measurement, probabilistic inference data-driven models

6 homogeneity and uniformity are impossible weather target selection hardware evolution efficiency considerations

7 probabilistic target selection SDSS-III SDSS-III BOSS quasar target selection in SDSS bandpasses, z 3 quasars look like A-type stars stars outnumber quasars enormously don t have good models of either Bovy et al. arxiv: this target selection cannot be uniform heterogeneous data quality means heterogeneous target selection star density varies on the sky suck it up!

8 homogeneity and uniformity are unnecessary correct the data compute inverse selection volume or probabilities 1/V max (ish) re-weight the data using these inverse volumes very wrong! forward modeling write down uncensored p0 (data parameters) multiply by (one minus) censoring rate η(data) renormalize to get expected p(data parameters) this is a likelihood function (but: visualizing a forward model)

9 estimators Cramèr Rao bound example: Gaia astrometry likelihood principle(s) it is our duty to analyze our very limited data with optimal methods the output of any data analysis must be a likelihood function WMAP, Planck

10 likelihood principle I said function. p(data parameters)

11 living the likelihood dream don t make a catalog of objects that s some kind of (probably inefficient) estimator even with error bars it can t transmit the full information produce a likelihood function in catalog space Lang et al. Brewer et al. arxiv:

15 homogeneity and uniformity are unnecessary? special case of two-point functions (and higher orders) currently an unsolved problem (but papers from Wandelt s group)

16 homogeneity and uniformity are harmful can t be uniform in everything the uniformity you choose only helps one of your customers! uniform samples end up requiring a lot of time on the least useful objects reduces the heterogeneity that is essential to calibration

17 self-calibration final imaging calibration of SDSS made no use at all of the calibration program data Padmanabhan et al. arxiv:astro-ph/

20 calibration programs are wasteful there are more photons in the science data therefore, the science data contain more information about calibration (exceptions abound) you must take your data with proper heterogeneity! Kepler tiling patterns Holmes et al. arxiv:

21 A 4 B Sky Position β (deg) C 4 D Sky Position β (deg) Sky Position α (deg) Sky Position α (deg)

1.0 Focal Plane Position y (deg) Focal Plane Position y (deg) 0.3 0.2 0.1 0.0 0.1 0.2 0.3 0.3 0.2 0.1 0.0 0.1 0.2 0.3 0.950 0.940 0.950 0.940 0.970 0.970 0.960 0.

0 0.8 0.6 0.4 0.2 0.0 0.2 0.4 0.6 0.8 Residuals (%) Focal Plane Position y (deg) Focal Plane Position y (deg) 0.3 0.2 0.1 0.0 0.1 0.2 0.3 0.3 0.2 0.1 0.0 0.1 0.2 0.3 0.910 0.

940 0.960 0.970 0.3 0.2 0.1 0.0 0.1 0.2 0.3 Focal Plane Position x (deg) (a) 0.890 (c) 0.890 0.3 0.2 0.1 0.0 0.1 0.2 0.3 Focal Plane Position x (deg) (b) (d) 1.

2 0.1 0.0 0.1 0.2 0.3 0.3 0.2 0.1 0.0 0.1 0.2 0.3 0.980 0.980 0.990 1.000 0.940 0.930 0.920 0.910 0.900 0.990 1.000 0.940 0.930 0.920 0.910 0.900 0.960 0.950 0.

8 0.6 0.4 0.2 0.0 0.2 0.4 0.6 0.8 1.0 Residuals (%) Focal Plane Position y (deg) Focal Plane Position y (deg) 0.2 0.1 0.0 0.1 0.2 0.3 0.3 0.2 0.1 0.0 0.1 0.2 0.3 0.990 0.

22 1.0 Focal Plane Position y (deg) Focal Plane Position y (deg) Focal Plane Position x (deg) (a) (c) Focal Plane Position x (deg) (b) (d) Residuals (%) Focal Plane Position y (deg) Focal Plane Position y (deg) Focal Plane Position x (deg) (a) (c) Focal Plane Position x (deg) (b) (d) Residuals (%) (a) (b) (a) (b) 1.0 Focal Plane Position y (deg) Focal Plane Position y (deg) Focal Plane Position x (deg) (c) Focal Plane Position x (deg) (d) Residuals (%) Focal Plane Position y (deg) Focal Plane Position y (deg) Focal Plane Position x (deg) (c) Focal Plane Position x (deg) (d) Residuals (%)

23 Self-calibration of imaging A good survey: every star appears in many images in different images, the star is in different places every image contains many stars Holmes et al. arxiv: Kepler and Spitzer exoplanet photometry is pessimal for self-calibration but for a very good reason!

24 target selection is classification SDSS-III SDSS-III BOSS is taking spectra of quasars, not stars stars outnumber (relevant) quasars by factors of hundreds observations are noisy and theoretical models are incomplete want to find only the quasars... or do we?

25 classification algorithms Support Vector Machine, Random Forest, Artificial Neural Net all bad! value of a causal model training and test samples don t match need to classify new data taken under different conditions make use of our technical knowledge about the data. Bovy et al. arxiv:

26 1-epoch model 30-epoch

30 aside: discovery as classification found an exoplanet? That s a model selection move. Bayes doesn t tell you how to make decisions. utility arises Make decisions that maximize expected (scientific?) return. Astrometry.net has an explicit utility model Automatic calibration of an image successful? Our customer model is that they are offended by false positives. Lang et al. arxiv:

31 utility considerations might be worth taking a source unlikely to be a quasar, as long as it is likely to be interesting need to be able to make these trade-offs quantitatively requires a specification of utility needs to be measured in dollars (or equivalent) long-term future discounted free cash flow the game of proposal writing we aren t honest in our proposals about what we want SDSS was over-designed by any measure that was valuable!

32 over-design SDSS was seriously over-designed to measure the large-scale structure (no-one thinks that was a bad idea) could have done all the large-scale structure in less than one year of observing we might have to be more honest going forward if we want to use resources efficiently, we need to face a trade-off between efficiency and discovery At the present, everything is heuristics. I say we make this trade-off explicitly, not implicitly.

33 utopia every part of your data analysis pipeline returns a likelihood function information propagation through the pipeline always by likelihood function implications are severe you can simulate data under different experimental designs likelihood is p(data parameters) you have a specified utility function converts information in your answer into dollars every decision can now be an optimization detectors, optical path, spectral elements filters, exposure times, cadences targets

34 example: bandpasses LSST plans to do imaging in ugrizy I am going to smash that r filter! why not do ugwizy? easy example because zero-cost change doesn t require full utility specification bet it is much better for low-s/n objects

36 hardware vs software trades P1640 Oppenheimer et al. arxiv: Fergus et al. in prep glitter cam Fergus et al. MIT-CSAIL-TR

42 open-source surveys Hipparcos example SDSS calibration example enormous benefits accrue from making the data re-reducable from scratch

43 throwing down the gauntlet Gaia uncertainty propagation (qualitative) Euclid observing strategy for imaging LSST bandpass, cadence, and exposure-time settings SKA pathfinder image products eboss two-point function estimators APOGEE & HERMES signal-to-noise requirements (My hourly rates are a bargain.) (These surveys are all awesome!)

44 punchlines Calibration programs are wasteful and reduce the accuracy of your end-of-mission results. (you will need to adjust your observing strategy) Homogeneity and uniformity of survey samples are impossible, unnecessary, and harmful goals. (you will need to implement some probability theory) Proper uncertainty propagation is not easy. (I got nothing) The challenge is to make precise measurements and keep discovery space open. (you will need to understand, quantitatively, your goals)

Designing Astronomical Projects for Real-Time Discovery, Operational Efficiency, and Statistical Re-Use of Legacy Data

Designing Astronomical Projects for Real-Time Discovery, Operational Efficiency, and Statistical Re-Use of Legacy Data Thank you Megan Bedell (Flatiron) Josh Bloom (Berkeley) Doug Finkbeiner (Harvard)