Data-Intensive Statistical Challenges in Astrophysics

Transcription

1 Data-Intensive Statistical Challenges in Astrophysics Collaborators: T. Budavari, C-W Yip (JHU), M. Mahoney (Stanford), I. Csabai, L. Dobos (Hungary) Alex Szalay The Johns Hopkins University

2 The Age of Surveys CMB Surveys (pixels) 1990 COBE Boomerang 10, CBI 50, WMAP 1 Million 2008 Planck 10 Million Angular Galaxy Surveys (obj) 1970 Lick 1M 1990 APM 2M 2005 SDSS 200M 2011 PS1 1000M 2020 LSST 30000M Time Domain QUEST SDSS Extension survey Dark Energy Camera Pan-STARRS LSST Galaxy Redshift Surveys (obj) 1986 CfA LCRS dF SDSS BOSS LAMOST Petabytes/year

3 Sloan Digital Sky Survey The Cosmic Genome Project Two surveys in one Photometric survey in 5 bands Spectroscopic redshift survey Data is public 2.5 Terapixels of images => 5 Tpx 10 TB of raw data => 120TB processed 0.5 TB catalogs => 35TB in the end Started in 1992, finished in 2008 Extra data volume enabled by Moore s Law Kryder s Law

4 Analysis of Galaxy Spectra Sparse signal in large dimensions Much noise, and very rare events 4Kx1M SVD problem, perfect for randomized algorithms Motivated our work on robust incremental PCA

5 Galaxy Properties from Galaxy Spectra Spectral Lines Continuum Emissions

6 Galaxy Diversity from PCA PC 1st [Average Spectrum] 2nd [Stellar Continuum] 3rd [Finer Continuum Features + Age] 4th 5th [Age] Balmer series hydrogen lines [Metallicity] Mg b, Na D, Ca II Triplet

7 Streaming PCA Initialization Eigensystem of a small, random subset Truncate at p largest eigenvalues Incremental updates Mean and the low-rank A matrix SVD of A yields new eigensystem Randomized algorithm! T. Budavari, D. Mishin 2011

8 Robust PCA PCA minimizes σ RMS of the residuals r = y Py Quadratic formula: Σr 2 extremely sensitive to outliers We optimize a robust M-scale σ 2 (Maronna 2005) Implicitly given by Fits in with the iterative method! Outliers can be processed separately

9 Eigenvalues in Streaming PCA 9 Classic Robust

10 Examples with SDSS Spectra Built on top of the Incremental Robust PCA Principal Component Pursuit (I. Csabai et al) Importance sampling (C-W Yip et al)

11 Principal component pursuit Low rank approximation of data matrix: X Standard PCA: min X E subject to rank ( E) k works well if the noise distribution is Gaussian outliers can cause bias Principal component pursuit sparse spiky noise/outliers: try to minimize the number of outliers while keeping the rank low NP-hard problem 2 min A 0 subjectto X = N + A, rank ( N) k The L1 trick: min ( N + λ A ) subjectto X = N + A * N, A 1 numerically feasible convex problem (Augmented Lagrange Multiplier) min N, A ( N λ A ) subjectto X ( N + A) < ε + * 1 2 * E. Candes, et al. Robust Principal Component Analysis. preprint, Abdelkefi et al. ACM CoNEXT Workshop (traffic anomaly detecion)

12 Testing on Galaxy Spectra Slowly varying coninuum + absorpion lines Highly variable sparse emission lines This is the simple version of PCP: the posiion of the lines are known but there are many of them, automaic detecion can be useful spiky noise can bias standard PCA DATA: Streaming robust PCA implementaion for galaxy spectrum catalog (L. Dobos et al.) SDSS 1M galaxy spectra Morphological subclasses Robust averages + first few PCA direcions

13 PCA PCA reconstrucion Residual

14 Principal component pursuit Low rank Sparse Residual λ=0.6/sqrt(n), ε=0.03

15 Galaxy ID Not Every Data Direction is Equal Wavelength Selected Wavelengths Wavelength A = C X Galaxy ID Selected Wavelengths Procedure: 1. Perform SVD of A = U Σ V T 2. Pick number of eigenvectors = K 3. Calculate Leverage Score = Σ i V T ij 2 / K Mahoney and Drineas 2009

16 Wavelength Sampling Probability k = 2 c = 7 k = 4 c = 16 k = 6 c = 25 k = 8 c = 29

17 Ranking Astronomical Line Indices Subspace Analysis of Spectra Cutouts: - Othogonality - Divergence - Commonality (Worthey et al. 94; Trager et al. 98) (Yip et al in prep.)

18 Identify Informative Regions NewMethod 1. Pick the λ with largest P λ 2. Define its region of influence using Σ λ P λ convergence. Mask λ s from future selection. 3. Go back to Step 1, or quit. MahoneySecond 1. Over-select λ s from the targeted number. 2. Merge selected λ if two pixels lie within a certain distance 3. Quit.

19 Identifying New Line Indices, Objectively (Yip et al in prep.)

20 New Spectral Regions (MahoneySecond; k = 5; Overselecting 10 X; Combining if < 30 Å)

21 NewMethod vs MahoneySecond NM M2

22 Gunawan & Neswan 2000)

23 Angle between Subspaces JHU Lick

24 Σ λ P λ JHU Lick

25 Line Indices for Galaxy Parameter Estimations

26 Importance Sampling and Galaxies Lick indices are ad hoc The new indices are objective Recover atomic lines Recover molecular bands Recover Lick indices Informative regions are orthogonal to each other, in contrast to Lick Future Emission line indices More accurate parameter estimation of galaxies

27 Summary Non-Incremental changes on the way Science is moving increasingly from hypothesisdriven to data-driven discoveries Need randomized, incremental algorithms Best result in 1 min, 1 hour, 1 day, 1 week New computational tools and strategies not just statistics, not just computer science, not just astronomy, not just genomics Astronomy has always been data-driven. now becoming more generally accepted