Data-Intensive Statistical Challenges in Astrophysics

Transcription

1 Data-Intensive Statistical Challenges in Astrophysics Alex Szalay The Johns Hopkins University Lots of contributions from Tamas Budavari (JHU)

2 Scientific Data Analysis Today Scientific data is doubling every year, reaching PBs Data is everywhere, never will be at a single location Architectures increasingly CPU-heavy, IO-poor Data-intensive scalable architectures needed Databases are a good starting point Scientists spend 80% of time on menial tasks Scientists need special features (arrays, GPUs) Most data analysis done on midsize BeoWulf clusters Universities hitting the power wall Soon we cannot even store the incoming data stream Not scalable, not maintainable

3 Non-Incremental Changes Science is moving increasingly from hypothesisdriven to data-driven discoveries Need new randomized, incremental algorithms Best result in 1 min, 1 hour, 1 day, 1 week New computational tools and strategies not just statistics, not just computer science, not just astronomy, not just genomics Need new data intensive scalable architectures Astronomy has always been data-driven. now becoming more generally accepted

4 Scalable Data-Intensive Analysis Large data sets => data resides on hard disks Analysis has to move to the data Hard disks are becoming sequential devices For a PB data set you cannot use a random access pattern Both analysis and visualization become streaming problems Same thing is true with searches Massively parallel sequential crawlers (MR, Hadoop, etc) Spatial indexing needs to be maximally sequential Space filling curves (Peano-Hilbert, Morton, )

5 Why Astronomy Data? Approach inherently and traditionally data-driven Important spatio-temporal features Very large density contrasts in populations Real errors and covariances Many signals very subtle, buried in systematics Data sets very large, pushing scalability Worthless! Jim Gray

6 The Age of Surveys CMB Surveys (pixels) 1990 COBE Boomerang 10, CBI 50, WMAP 1 Million 2008 Planck 10 Million Angular Galaxy Surveys (obj) 1970 Lick 1M 1990 APM 2M 2005 SDSS 200M 2011 PS1 1000M 2020 LSST 30000M Time Domain QUEST SDSS Extension survey Dark Energy Camera Pan-STARRS LSST Galaxy Redshift Surveys (obj) 1986 CfA LCRS dF SDSS BOSS LAMOST Petabytes/year

7 Sloan Digital Sky Survey The Cosmic Genome Project Two surveys in one Photometric survey in 5 bands Spectroscopic redshift survey Data is public 2.5 Terapixels of images => 5 Tpx 10 TB of raw data => 120TB processed 0.5 TB catalogs => 35TB in the end Started in 1992, finished in 2008 Extra data volume enabled by Moore s Law Kryder s Law

8 Astro-Statistical Challenges The crossmatch problem (multi-, time domain) Photometric redshifts (prediction/regression problem) Correlations (auto/cross, higher order) Outlier detection in many dimensions Statistical errors vs systematics Comparing observations to models comparing distributions, updating models The unknown unknown, when we have no models.. Exascale streams are coming (Square Km Array) Scalability!!!

9 Relevant Astro Challenges I. Analysis of spectra (multispectral data) Sparse signal in large dimensions Much noise, and very rare events 4Kx1M SVD problem, perfect for randomized algorithms Motivated our work on robust incremental PCA Photometric measurements (segmentation) Currently 500M rows x 500 attributes Noisy, missing data Spatial, temporal searches and associations Dominated by unknown systematic errors Small number of subtle outliers, buried in very large typical population Very skewed distributions

10 Galaxy Diversity from PCA PC 1st [Average Spectrum] 2nd [Stellar Continuum] 3rd [Finer Continuum Features + Age] 4th 5th [Age] Balmer series hydrogen lines [Metallicity] Mg b, Na D, Ca II Triplet

11 Streaming PCA Initialization Eigensystem of a small, random subset Truncate at p largest eigenvalues Incremental updates Mean and the low-rank A matrix SVD of A yields new eigensystem Randomized algorithm! T. Budavari, D. Mishin 2011

12 Robust PCA PCA minimizes σ RMS of the residuals r = y Py Quadratic formula: r 2 extremely sensitive to outliers We optimize a robust M-scale σ 2 (Maronna 2005) Implicitly given by Fits in with the iterative method! Outliers can be processed separately

13 Eigenvalues in Streaming PCA 13 Classic Robust

14 Examples with SDSS Built on top of the Incremental Robust PCA Principal Component Pursuit (I. Csabai) CX decomposition (C-W Yip)

15 Principal component pursuit Low rank approximation of data matrix: X Standard PCA: min X E subject to rank( E) k 2 works well if the noise distribution is Gaussian outliers can cause bias Principal component pursuit min subjectto sparse spiky noise/outliers: try to minimize the number of outliers while keeping the rank low NP-hard problem A 0 X N A, rank( N) k The L1 trick: min N A subjectto X N A * N, A 1 numerically feasible convex problem (Augmented Lagrange Multiplier) min N, A N A subjectto X ( N A) * 1 2 * E. Candes, et al. Robust Principal Component Analysis. preprint, Abdelkefi et al. ACM CoNEXT Workshop (traffic anomaly detection)

16 Testing on Galaxy Spectra Slowly varying continuum + absorption lines Highly variable sparse emission lines This is the simple version of PCP: the position of the lines are known but there are many of them, automatic detection can be useful spiky noise can bias standard PCA DATA: Streaming robust PCA implementation for galaxy spectrum catalog (L. Dobos et al.) SDSS 1M galaxy spectra Morphological subclasses Robust averages + first few PCA directions

17 PCA PCA reconstruction Residual

18 Principal component pursuit Low rank Sparse Residual λ=0.6/sqrt(n), ε=0.03

19 Galaxy ID Galaxy ID Not Every Data Direction is Equal Wavelength Selected Wavelengths Wavelength A = C X Selected Wavelengths Procedure: 1. Perform SVD of A = U V T 2. Pick number of eigenvectors = K 3. Calculate Leverage Score = i V T ij 2 / K Mahoney and Drineas 2009

20 Wavelength Sampling Probability k = 2 c = 7 k = 4 c = 16 k = 6 c = 25 k = 8 c = 29

21 Ranking Astronomical Line Indices Subspace Analysis of Spectra Cutouts: -Othogonality -Divergence -Commonality (Worthey et al. 94; Trager et al. 98) (Yip et al in prep.)

22 Identifying New Line Indices, Objectively (Yip et al in prep.)

23 Relevant Astro Challenges II. Photometric redshifts High dimensional embedding of a noisy low-dimensional process: guess galaxy distance from its colors Subtle statistical problem, with huge computational needs => scalability challenge Training set => regression/estimation engine Models constrained by Bayesian priors during training (Budavari) Need to detect atypical objects that do not fit known patterns Large-Scale Fuzzy Spatial Cross-Matching Soon we need to do spatial matches 100Bx100B data sets Experimenting with different approaches (DB/Hadoop/GPU)

24 Photometric Redshifts Normally, distances from Hubble s Law H r Measure the Doppler shift of spectral lines distance! v 0 But spectroscopy is very expensive SDSS: 640 spectra in 45 minutes vs. 300K 5 color images Future big surveys will have no spectra Idea: Multicolor images are like a crude spectrograph Statistical estimation of the redshifts/distances

25 Photometric Redshifts Phenomenological (PolyFit, ANNz, knn, RF) Simple, quite accurate, fairly robust Little physical insight, difficult to extrapolate, Malmquist Template-based (KL, HyperZ ) Simple, physical model Calibrations, templates, issues with accuracy Hybrid ( base learner ) Physical basis, adaptive Complicated, compute intensive Important for next generation surveys! We must understand the errors!

26 Recent Developments Random Forests S. Carliles, C. Priebe, A. Szalay, T. Budavari, S. Heinis (2009) Slightly better than other estimators Estimated errors close to Gaussian, and accurate Physically motivated removal of various systematics Inclination Self Absorption in a galaxy (Yip et al 2011) Effect of emission lines Unified theory of photometric redshifts (Budavari 2010) Not a regression problem Kernel density estimators, constrained by model priors

27 RF is like Kernel Estimator For a given query point each tree picks a neighborhood The ensemble of these neighborhood points defines a kernel The smaller the sampling rate from the training set, the broader the kernel => effective degrees of freedom N eff Central limit theorem convergence is driven by N eff RF is computationally a very efficient way to regress

28 Carliles et al 2009 Zspec vs Zrf

29 RF on Cyberbricks 36-node Amdahl cluster using 1200W total Zotac Atom/ION motherboards 4GB of memory, N330 dual core Atom, 16 GPU cores Aggregate disk space 43.6TB 63 x 120GB SSD = 7.7 TB 27x 1TB Samsung F1 = 27.0 TB 18x.5TB Samsung M1= 9.0 TB Blazing I/O Performance: 18GB/s Amdahl number = 1 for under $30K Using the GPUs for data mining: 6.4B multidimensional regressions (photo-z) in 5 minutes over 1.2TB of data Running the Random Forest algorithm inside the DB

30 Correlation Functions 2-point, 3 point 2 nd order, higher order Spatial, angular Various estimators Direct estimators, edge corrections (Ripley, Landy-Szalay) Various algorithms 1 2 Tree codes (a. Gray) GPU-based codes (Budavari)

31 The Impact of GPUs We need to reconsider the N logn only approach Once we can run 100K threads, maybe running SIMD N 2 on smaller partitions is also acceptable Recent JHU effort on integrating CUDA with SQL Server, using SQL UDF Galaxy spatial correlations: 600 trillion real and random galaxy pairs using brute force N 2 Much faster than the tree codes! This is because high resolution was needed Tian, Budavari, Neyrinck, Szalay 2010

32 Visualizing Petabytes Visualizations are becoming IO limited Needs to be done where the data is It is easier to send a HD 3D video stream to the user than all the data It is possible to build individual servers with extreme data rates (5GBps per server see Data-Scope) Prototype on turbulence simulation already works: data streaming directly from SQL Server to GPU Interactive visualizations driven remotely

33 Streaming Visualization Kai Buerger (visited JHU from Munich) Implemented high performance streaming from DB into the GPU, exceeding 1GB/sec Limited by the current I/O subsystem Using the velocity field from our 30TB turbulence database

34

35

36 JHU Data-Scope Revised 1P 1S All P All S Full servers rack units capacity TB price $K power kw GPU* TF seq IO GBps IOPS kiops netwk bw Gbps * Without the GPU costs (it is about $1,600/ card)

37 Summary Real multi-pb solutions are needed NOW! We have to build it ourselves Multi-faceted challenges No single trivial breakthrough, will need multiple thrusts Various scalable algorithms, new statistical techniques Adoption of new approaches is a difficult process Need to train people with new skill sets Π shaped people Increasing diversification Also many common patterns across all disciplines A new, Fourth Paradigm of science is emerging but it is not incremental.