Censoring and truncation in astronomical surveys. Eric Feigelson (Astronomy & Astrophysics, Penn State)

Size: px

Start display at page:

Download "Censoring and truncation in astronomical surveys. Eric Feigelson (Astronomy & Astrophysics, Penn State)"

Norma Ramsey
5 years ago
Views:

1 Censoring and truncation in astronomical surveys Eric Feigelson (Astronomy & Astrophysics, Penn State)

2 Outline 1. Examples of astronomical surveys 2. One survey: Truncation and spatial/luminosity distributions 3. Two or more surveys: Censoring and luminosity distributions & relationships 4. A call to arms

3 Faint Images of the Radio Sky at Twenty-centimeters (FIRST) 811,000 radio sources

4 Infrared Astronomical Satellite (IRAS) Early 1980s 350,000 mid-ir sources in all-sky survey

5 Two-Micron All Sky Survey (2MASS) Early 2000s 300,000,000 stars and galaxies

6 Sloan Digital Sky Survey (SDSS) 2000s 100,000,000 visual band stars and galaxies plus 1,000,000 spectra

7 ROSAT All-Sky Survey (RASS) 100,000 X-ray sources plus diffuse emission

8 Energetic Gamma-Ray Experiment Telescope (EGRET) on the Compton Gamma-Ray Observatory 1990s 271 sources with E>100 MeV

Star counts: The first flux limited surveys For a uniform population of objects distributed randomly in transparent space: S = L / 4 π D 2 V = 4/3 π D 3 log N ~ log V ~ -1.

9 Star counts: The first flux limited surveys For a uniform population of objects distributed randomly in transparent space: S = L / 4 π D 2 V = 4/3 π D 3 log N ~ log V ~ -1.5 log S William Herschel (1785) used deviations from this prediction to infer that the Universe (now known as our Milky Way Galaxy) is limited in extent (~1 kpc) and is elongated in shape.

10 Star counts at different Galactic latitudes halo disk The Galaxy has multiple components with different spatial distributions producing strong deviations from the logn~-1.5 logs law. Bahcall & Soneira 1980 The universe at faint magnitudes. I - Models for the galaxy and the predicted star counts

11 But, near the Galactic plane, space is very opaque due to interstellar dust. Thus, Herschel s galaxy is ~20 times too small, which was not appreciated until the s. Big effort in s to address the Fundamental equation of stellar statistics to simultaneously establish the distribution of stellar luminosities and their spatial distributionin the Galactic disk: A(S,l,b) = Int [D(r,l,b) Φ(L,Abs(r,l,b)) r 2 dr A = number of stars/deg -2 in flux interval around S towards direction (l,b). (Star count data) D = number of stars/pc -3 at distance r from Earth towards (l,b) Φ = normalized distribution of stellar luminosities (LF) corrected for spatially-dependent absorption Fredholm Type 1 integral equation solved by Taylor expansion, Fourier transform, numerical integration. This effort essentially failed in the disk due to the patchy absorption, but functions reasonably well towards the Galactic poles (b=90 o ). Trumpler & Weaver Statistical Astronomy 1953

12 Stellar populations & kinematics in the Galaxy New catalogs with 4-6 dimensions of phase space: (α,δ) Location in the sky π Parallax giving distance (µ α, µ δ ) Proper motion across the sky V r Radial velocity 50,000 stars have all 6 values measured, 100,000 have 5 values, 30M have 4 values (Hipparcos, UCAC2, Gen-Cop, RAVE, ). Fascinating structure: Thin disk, thick disk, halo components of Galaxy Trace star formation around Sun Halo streams & cannibalized galaxies Very little sophisticated statistical work has been performed on these catalogs (e.g. mixture models, wavelet analysis)

13 Model of Milky Way cannibalizing the Sagittarius galaxy

14 Extragalactic surveys: The simple case of nearby galaxies

15 A recent SDSS study Volume-limited survey of 28,089 with d<150 Mpc Low lum High lum Blanton et al The Properties and Luminosity Function of Extremely Low Luminosity Galaxies

16 Concentration Surface brightness Color Luminosity

17 Normal galaxy luminosity function Parametric model: Schechter function = Gen. Pareto distribution dn/dl ~ Φ ~ L -α e -L/L* Nonparametric methods: two used here (1/V max, stepwise max. likelihood) (Many technical issues concerning correction for missing low-surface brightness galaxies, K-correction and evolution-corrections to luminosity, double Schechter fits, etc.)

18 Nonparametric estimators to LFs 1. Classic estimator: Φ(L) = N(L) / V where N is the number of stars/galaxies/agns in surveyed volume V. A biased estimator. 2. Schmidt estimator (>700 citations): Φ(L) = Σ 1 / V max (L i ) where V max is the maximum volume within which an object of the observed flux could have been seen given the survey s sensitivity limit. Unbiased estimator but with high variance. Schmidt 1968 Space Distribution and Luminosity Functions of Quasi-Stellar Radio Sources Felten 1976 On Schmidt's V m estimator and other estimators of luminosity function

Lynden-Bell 1971 A method of allowing for known observational selection in small

19 Lynden-Bell-Woodroofe MLE (>100 citations): Recursion relation: where C k is the number of stars/galaxies/agns in the k-th rectangle in the luminosity-distance diagram Lynden-Bell 1971 A method of allowing for known observational selection in small samples applied to 3CR quasars Woodroofe 1985 Estimation of a distribution function with truncated data

20 Stepwise maximum-likelihood estimator (>500 citations): where Set dl/dφ = 0 to obtain Efstathiou et al Analysis of a complete galaxy redshift survey. II - The field-galaxy luminosity function

21 Takeuchi et al. (2000) apply these LF estimators to Monte Carlo simulations of small galaxy catalogs (N=100 and 1000). All perform well when spatial distribution is homogeneous. But for spatially clustered distributions, the 1/V max estimator is badly affected. There has been no analytical evaluation of the mathematical properties of these estimators since study of 1/V max by Felten (1976).

22 Extragalactic surveys The difficult case of quasar evolution AGN exhibit huge cosmic evolution: hundreds of times more luminous quasars when the Universe was half its current age compared to today.

23 Here is the logn-logs distribution of radio sources in the sky. The expected logn~-1.5logs of a uniform distribution has been removed here, so deviations from a horizontal line indicates cosmic evolution of the volume density and/or LF. The radio sources arise from two main classes, AGN and star-forming galaxies, which exhibit different evolution behaviors. Enormous literature s on quasar evolution, and a s literature on star formation evolution (the `Madau plot ). Condon et al Radio sources and star formation in the local universe

24 Two or more flux limited surveys 1. A flux-limited survey gives a catalog of sources at one waveband 2. Astronomers observe this sample at another waveband, but only some objects are detected 3. Questions: What is the LF in the second waveband? (density estimation) Are these LFs the same in different subsamples? (k-sample test) What is the relationship between the first and second luminosities? (correlation & regression) 4. Generalize to a multivariate database with many properties measured for the initial sample. Nondetections appear in many columns. In astronomical parlance, the nondetections give upper limits. In statistical parlance, the nondetections give left-censored data points. These questions have arisen in hundreds of studies

25 An example of astronomical censored data Heckman et al A millimeter-wave survey of CO emission in Seyfert galaxies

26 Several solutions to censored LF problem were considered by qstronomers during , leading to the rediscovery of Efron s redistribute-to-the-right algorithm for the maximum-likelihood Kaplan-Meier product-limit estimator for randomly censored data. Schmitt, Feigelson & Nelson and Isobe et al. introduced astronomers to existing methods of survival analysis for treatment of censored data. Schmitt 1985 Statistical analysis of astronomical data containing upper bounds Feigelson & Nelson 1985 Statistical methods for astronomical data with upper limits I - Univariate distributions Isobe et al Statistical methods for astronomical data with upper limite II - Correlation and regression

27 Example of univariate survival analysis (Kaplan-Meier) Feigelson & Nelson 1986

28 Test of bivariate correlation/regression for simulated flux-limited surveys Biased line using detections only Isobe, Feigelson & Nelson 1986 Unbiased Buckley-James line Including nondetections

29 ASURV code Stand-alone code implementing survival methods for astronomy: Kaplan-Meier estimator for the luminosity function Gehan, logrank, Peto-Peto & Peto-Prentice 2-sample tests Cox regression for bivariate correlation Generalized Kendall s τ & Spearman s ρ coefficient including censoring in both variables (Brown, Hollander & Korwar, Akritas) Linear regression with Gaussian residuals (EM Algorithm) Linear regression with K-M residuals (Buckley-James) Linear regression with censoring in both variables (Campbell, Schmitt) LaValley, Isobe & Feigelson 1992 ASURV software report

Methodological work needed for astronomical censored data 1 Investigate best methods for astronomical censoring patterns Type 1 censoring in F=L/4pd 2 gives

30 Methodological work needed for astronomical censored data 1 Investigate best methods for astronomical censoring patterns Type 1 censoring in F=L/4pd 2 gives nonrandom censoring in L 2. Develop multivariate analysis for censoring in all variables Clustering, regression/pca, MANOVA, MARS & neural nets, etc. Shapley et al. 2001

31 3. Calculate nonlinear regression & goodness-of-fit 4. Treat simultaneous censoring & truncation Astronomical survey Goal: Characterizing LFs and relationships in stars/gals/agns Full population of stars/galaxies/agns Truncated sample from fluxlimited survey Censored measurements of other properties AIDS epidemic Goals: Characterizing infected populations & mortality from AIDS Full population of infected individuals Truncated sample from symptomatic individuals Censored measurements of survival 5. Treat simultaneous censoring & measurement error Unlike biometrical and industrial reliability testing environments where survival times are measured precisely, in astronomy the censored value is typically set at the 3σ upper limit where σ is the known noise level. A new statistical approach is needed that treats all observations (detected or not) with measurement errors. Fisher s fiducial distributions??

32 A call to arms Astrostatistical research for astronomical surveys Develop mathematics for nonparametric estimators for luminosity functions in flux-limited surveys, including applications to complex situations like spatial inhomogeneities & cosmic evolution. Perform state-of-the-art multivariate/spatial analysis of Galactic stellar populations using Hipparcos/UCAC2/ databases Extend survival methods to truncated, multiplycensored, measurement-error, multivarate databases

Censoring and truncation in astronomical surveys

Censoring and truncation in astronomical surveys Eric Feigelson Statistical Modeling of Cosmic Populations Penn State Summer School June 2014 Selection bias in flux-limited surveys Astronomers often struggle