SKA Science Data Processing or MAKING SENSE OF SENSOR BIG DATA

Transcription

1 SKA Science Data Processing or MAKING SENSE OF SENSOR BIG DATA Bojan Nikolic & SDP Consortium Astrophysics Group, Cavendish Laboratory Project Engineer/Architecture Lead SKA Science Data Processor

2 What is the Square Kilometre Array (SKA)? The Next Generation radio telescope compared to best current instruments it is...

3 What is the Square Kilometre Array (SKA)? The Next Generation radio telescope compared to best current instruments it is... ~100 times sensitivity ~ 10 6 times faster imaging the sky More than 5 square km of collecting area on sizes up to 3000km JVLA 27 27m dishes Longest baseline 30km GMRT 30 45m dishes Longest baseline 35 km

4 What is the Square Kilometre Array (SKA)? The Next Generation radio telescope compared to best current instruments it is... ~100 times sensitivity ~ 10 6 times faster imaging the sky More than 5 square km of collecting area on sizes up to 3000km Will address some of the key problems of astrophysics and cosmology (and physics) Builds on techniques developed here in the Cambridge It is an Aperture Synthesis radio telescope ( interferometer ) Uses innovative technologies... Major ICT project Need performance at low unit cost Phased deployment This talk only about the first phase SKA1 due for full operations in 2022

5 SCIENCE DRIVERS

6 Epoch of Re-ionization Furlanetto et al. (2003) z= Mpc comoving Dn=0.1 MHz

7 Epoch of Re-ionization Furlanetto et al. (2003) z= Mpc comoving Dn=0.1 MHz

8 Furlanetto et al. (2003) Epoch of Re-ionization z= Mpc comoving Dn=0.1 MHz

9 Furlanetto et al. (2003) Epoch of Re-ionization z= Mpc comoving Dn=0.1 MHz

10 Furlanetto et al. (2003) Epoch of Re-ionization z= Mpc comoving Dn=0.1 MHz

11 Furlanetto et al. (2003) Epoch of Re-ionization z= Mpc comoving Dn=0.1 MHz

12 Furlanetto et al. (2003) Epoch of Re-ionization z= Mpc comoving Dn=0.1 MHz

13 Furlanetto et al. (2003) Epoch of Re-ionization z= Mpc comoving Dn=0.1 MHz

14 Furlanetto et al. (2003) Epoch of Re-ionization z= Mpc comoving Dn=0.1 MHz

15 Furlanetto et al. (2003) Epoch of Re-ionization z= Mpc comoving Dn=0.1 MHz

16 Furlanetto et al. (2003) Epoch of Re-ionization z= Mpc comoving Dn=0.1 MHz

17 Furlanetto et al. (2003) Epoch of Re-ionization z= Mpc comoving Dn=0.1 MHz

18 Furlanetto et al. (2003) Epoch of Re-ionization z= Mpc comoving Dn=0.1 MHz

19 Furlanetto et al. (2003) Epoch of Re-ionization z= Mpc comoving Dn=0.1 MHz

20 Pulsar Surveys: Testing gravity The SKA will detect around 30,000 pulsars in our own galaxy, 2000 msec pulsars accurate clocks Relativistic binaries give unprecedented strongfield test of gravity expect ~100 Timing net of ms pulsars to detect gravitational waves via timing residuals Expect timing accuracy to improve by ~100

21 Pulsar timing array Nano-Hertz range of frequencies MBH-MBH binaries: resolved objects and stochastic background Cosmic strings and other exotic phenomenon Timing residual ~10s ns need ms pulsars

22 Find the unexpected Hubble Deep Field (HDF) Very Large Array observation of HDF ~ 3000 galaxies ~ 3000 galaxies ~15 radio sources

23 Find the unexpected (2) Hubble Deep Field (HDF) Simulation of SKA Observation ~ 3000 galaxies

24 What are we trying to find/measure? Gravitational waves ( ~ nanohz signal) Structure in the pristine primordial gas due to the first galaxies forming (> Mpc signal) Evolution of structure of interstellar gas in galaxies over the second half of the Universe s age One-off flashes in radio and use them for Cosmology Imprint of primordial fluctuations on the distribution of galaxies in the nearby Universe The history of formation of stars in the Universe Dust -> pebbles -> boulders -> planets

25 THE SKA OBSERVATORY

26 The SKA Observatory SKA 1 SKA1-Low: Aperture Array, 1000 stations each with 256 antennas SKA1-Mid: 250 dishes with single pixel receivers SKA1-Survey: 96 dishes with phased array receivers each forming 36 beams Computers! Digital signal processing & general purpose computing

27 SKA Context Diagram Monitor and Control SKA1 Low: Low Frequency Aperture Array SKA1 Survey: Dish Antenna s with PAFs LFAA Correlator/ Beam Former Survey Correlator Science Data Processor Implementation (Australia) SKA1 Mid: Dish Antennas with Single- Pixel feeds SKA1 Mid Correlator/ Beam Former Pulsar Search Processor Science Data Processor Implementati on (South Africa) Astronomers

28 Basics of Interferometry Astronomical signal (EM wave) s B. s Visibility: Detect & amplify Digitise & delay Correlate Integrate 1 B 2 X X X X X X V(B) = E 1 E 2 * = I(s) exp( i w B.s/c ) Resolution determined by maximum baseline q max ~ l / B max Field of View (FoV) determined by the size of each dish q dish ~ l / D Process ( == SDP) Calibrate, grid, FFT SKY Image

29 Large D vs Large N GBT 100-m diameter telescope SKA LFAA prototype array No 1 aim: collect as many photons as possible No 2 aim: maximum separation of collectors -> achieve high angular resolution

30 Sampling - Each pair of telescopes results in a measured visibility - Loosely equivalent to a sample in the Fourier domain - Inevitably areas without any samples: - At high radius -> limit of attainable resolution - At small radius -> limit of largest detectable structure - Regular distribution of telescopes leads to a regular distribution of samples in uv plane Illustrative example for the 27 antenna JVLA!

31 Sampling - Each pair of telescopes results in a measured visibility - Loosely equivalent to a sample in the Fourier domain - Inevitably areas without any samples: - At high radius -> limit of attainable resolution - At small radius -> limit of largest detectable structure - Regular distribution of telescopes leads to a regular distribution of samples in uv plane Preliminary example for the 250 antenna SKA1-Mid

32 Sampling - Each pair of telescopes results in a measured visibility - Loosely equivalent to a sample in the Fourier domain - Inevitably areas without any samples: - At high radius -> limit of attainable resolution - At small radius -> limit of largest detectable structure - Regular distribution of telescopes leads to a regular distribution of samples in uv plane Illustrative example for the 27 antenna JVLA!

33 Sampling - Earth rotation effective moves the sampling point of each pair of telescopes - This limits the possible integration time duration - Fills in the gaps between the previous samples (but not the central gap and outside the limit) Illustrative example for the 27 antenna JVLA!

34 Sampling - More rotation Illustrative example for the 27 antenna JVLA!

35 Redundant Sampling With SKA1 there will be very many individual visibilities in each observation: Accumulation of visibilities improves the signal/noise Fills in most of the uv plane In general however to retain best signal/noise significant unevenness in sampling of the uv plane must be accepted. -> Deconvolution Illustrative example for the 27 antenna JVLA!

36 DATA PROCESSING REQUIREMENTS

37 Key Requirements for SDP Solve for: Image of the sky Effects of the atmosphere, imperfect sampling and imperfect telescope mechanics/electronics ( calibration ) In order to: Find/measure very faint signals Correct for some of the atmospheric/mechanical problems in real time Find transient phenomena Produce science-ready data products

38 Structure of input data Raw input data labelling Predictable & regular Natural labelling Predictable, largely regular Time 2Hz 20 Hz Up to 6 hours (Time) Only for transient phenomena Beam SKA1-Survey 36 beams Pointing Centre Constant for duration of observation Multi-beams folded into this dimension Frequency Channel 256k channels Frequency Channel Polarisation Products 4 products Polarisation Product Baseline 500k baselines uvw coordinates Visibility value ~16 bytes Visibility value ~16 bytes

39 Data-parallelism schemes Time & baseline Frequency Visibility data Exploit frequency independence o Data parallelism: Dominated by frequency o Provides dominant scaling o Nothing more needed if each processing node can manage a frequency channel complete processing Grid and de-grid Buffered UV data FFT Processing nodes

40 Data-parallelism schemes Time & baseline Frequency FFT Visibility data Exploit frequency independence o o o o Sort and distribute visibility data and target Grid and de-grid Gather target grids Further data parallelism in locality in UVW-space Use to balance memory bandwidth per node Some overlap regions on target grids needed UV data buffered either on a locally shared object store of locally on each node

41 Data Parallelism Schemes Time & baseline Frequency Visibility data FFT o o Exploit frequency independence Distribute visibility data, duplicate target Grid and de-grid Gather and accumulate target grids To manage total I/O from buffer/bus distribute Visibility data across nodes for same target grid which is duplicated Duplication of target provides fall-over protection

42 Solving for the image of the Sky & Calibration Uncalibrated Offset Calibration Rick Perley & Oleg Smirnov: High Dynamic Range Imaging,

43 Solving for the image of the Sky & calibration Self-Calibration closure error calibration Rick Perley & Oleg Smirnov: High Dynamic Range Imaging,

44 Illustrative problem sizing Input Data About 5 TeraByte/s input stream Process about 6 hours of data together -> 100 PetaBytes 256k Frequency Channels x 4 polarisation products x 500k baselines x 36k time samples Image Reconstruction About operations per byte of input Iterate through the dataset about 10 times Need about 100 PetaFLOP/second Output Data size ~ 10k x 10k pixels x 30k frequency channels x 4 Stokes x 8 bytes 100 TeraByte of output per observation (6 hours) How much to store? How to get to astronomers? How much re-processing of this data is feasible?

45 SDP Computational Requirements Model Compute Island Many Core Processor/ Accellerator Rcu,FLOP Rcu,bw Fast Working Memory Mcu,work Visibility Buffer Mcu,buf Slower pool memory Mcu,pool Visibility Buffer Mcu,buf Infiniband Interconnect

46 Maximal Case Design Equations Design SKA1-Low SKA1-Mid (Band 1) SKA1-Survey (Band 1) Equation 1 N cu R cu,flop > 102 PetaFLOP N cu R cu,flop > 361 PetaFLOP N cu R cu,flop > 135 PetaFLOP 2 N cu R cu,bw > 204 PetaByte N s cu R cu,bw > 722 PetaByte N s cu R cu,bw > 270 PetaByte s 3 N cu M cu,buf > 470 PetaByte N cu M cu,buf > 220 PetaByte N cu M cu,buf > 295 PetaByte 4 N cu R cu,io > 58 TeraByte s N cu R cu,io > 30 TeraByte s N cu R cu,io > 26 TeraByte s 6a M cu,work > 37 GigaByte M cu,work > 100 GigaByte M cu,work > 50 GigaByte 6b M cu,pool > 370 GigaByte M cu,pool > 1 TeraByte M cu,pool > 500 GigaByte 6c R cu,flop > 132 GigaFLOP s R cu,flop > 1600 GigaFLOP s Document SKA-TEL-SDP , Table 9, version of 15/9/2014 R cu,flop > 171 GigaFLOP s

47 Parametric Model / Design Equations

48 Non-performance requirements Data analysis scheduled together with data taking: Data analysis must complete in bounded time Must be able to pre-calculate the time-bound before seeing the data High Availability Sensitivity Success often defined in terms of Signal/Noise ratio in output data Some Streaming Processing: Find transients Feedback of initial calibration solutions to rest of the system Main computational load are (slightly) iterative algorithms: Imaging and Calibration Usual software reqs: maintainability, long-term sustainability, applicability to new computing hardware architectures and algorithms

49 PRELIMINARY ARCHITECTURE

50 Preliminary Architecture -- Hardware Homogeneous cluster of commodity nodes with internal heterogeneity: Low latency processors + accelerators (+ NUMA + multiple IB cards) High degree of multi-threaded parallelism: > 2 TeraFLOPs deployed on each working memory unit Nodes organised into (rack-sized) islands; high bisectional bandwidth within island but limited bisectional bandwidth outside islands Distributed high-performance storage for a scratch buffer Not shared between islands Probably object based

51 Data Ingress Architecture

52 Interconnect Inter-Island Interconnect concept: InfiniBand 8:1 oversubscribed Three levels of switches Up to ~35000 nodes Island Island Island Island

53 Software overview of programming model Dataflow programming model at the coarsest level Fully distributed control-flow: every island individually knows all of the operations it needs to do No data-dependent control flow Scheduling of data movement as critical as scheduling of computation Statically schedule the dataflow program down to the island level Rough estimate: 100 million dataflow tokens, 1GigaByte each High-availability & dynamic scheduling at island level Failover between nodes Work stealing within a node Massively multi-threaded shared-memory parallelism at socket level Thread scheduling to optimise memory access and instruction issue

54 Software components Pipeline descriptions the dataflow program Local Monitor & Control the top-level run-time system, organising all communication with the rest of the observatory Dataflow manager computes the optimised static schedule of the dataflow program Data manager: Implement data movements between different address spaces Implement data splitting/combining/filtering/binning/sorting Computational Actors: Stateless computational kernels implementing the mathematics of radio astronomy aperture synthesis Distributed, replicated configuration/metadata/parameter database Updated after each iteration through whole dataset

55 Software high availability Classify arcs in the dataflow graph as precious or nonprecious Precious data are treated in usual way failover, RAID, etc. Non-precious data can be dropped: If they are input to a map-type operation then no output If they are input to a reduction then result is computed without them Stragglers outputting non-precious data can be terminated after a relatively short time-out

56 Timeline 01/02/2012 SDP Conceptual Design Review 21/03/2015 Preliminary Design Review Today 01/09/2016 SDP Critical Design Review 21/12/2019 Begin Early Science 30/07/ /12/2021 SDP Construction: hardware roll out & software engineering 30/12/ /12/2022 Full Science Operations 01/01/ /01/ /01/ /01/ /01/ /01/ /01/ /01/ /01/ /01/ /01/ /01/ /12/ /03/ /11/2013 SDP Consortium Organisation 01/11/ /09/2016 SDP Preconstruction 30/09/ /07/2017 SDP Procurement Exercise? 01/03/ /11/2013 SDP Consortium Organisation 01/11/ /09/2016 SDP Preconstruction Today 01/01/ /04/ /07/ /10/ /01/ /04/ /07/ /10/ /12/ /11/ /06/2013 SDP Proposal Submission 04/11/2013 SDP Kickoff 01/08/2014 M6 Preliminary Architecture 15/09/2014 Milestone 7: Rebaselining SDP Consortium been going for almost 1 year now Important milestones coming up soon Many more years of design/construction until full operations

57 BIG COMPUTE / BIG DATA Sensor?

58 Big Compute -- Quantum Chemistry Calculations Density Functional Theory computation of frontier molecular orbitals, Jaiswal et al, 2013, doi:// /c3ra45806g

59 Big Compute -- Simulations Furlanetto et al. (2003) z= Mpc comoving Dn=0.1 MHz

60 Big Data -- Internet Big Data L. Page, 1999, ilpubs.stanford.edu/422/1/ pdf

61 Sensor Big Data Radio astronomy Remote sensing Earth observation Geophysics Medical (imaging or other) Internet of Things? 10 Hz sampling of the air temperature inside your fridge?

62 A Classification? Application Input Data Volume Input Information Density Computational Intensity Spreadsheet Low High Low N/A HPC Simulation Low High High High Conventional Big Data Algorithm Message Rate High High High Map-Reduce: Low Graph Problems: High Sensor Big Data High Low Intermediate Low

63 Characteristics of sensor big data Noisy Information content << sampling rate Combination/correlation of data necessary to find signals of interest -> volume! Multiple Streams of Input Data Calibration effects Incomplete/ imperfect sampling Volume/shape/ characteristics of data known in advance High degree of inherent parallelism at the front-end of processing Large sensors networks can not be made perfect Allow biases in measurement and find and correct for these in post-processing Poor control over experiment Expensive to precisely specify the sampling points Non-parametric statistics from incomplete/imperfect data

64 Summary of SKA Science Data Processor Very high rate of flow of data, intermediate computational intensity High degree of data parallelism in early stages of processing Statically schedulable algorithms at the large scales Iterative algorithm but with few iterations, relatively low intrinsic message rates High availability by discarding non-precious data/tasks Toward a solution by combining HPC and datacentre technologies

65 Challenges Power and capital expenditure if you try to extract all the information all the time Difficult to see an implementation that does not need to combine massive multi-threaded parallelism as well as massive multi-task parallelism Software sustainability/maintainability/ extensibility (8 years to beginning of full operations, min 30 yrs to end) Coordinating construction between elements of the observatory telescopes, infrastructure, digital signal processing, signal transport, science data transport) Balance between short-term and long-term risks Prising people away from the current programing paradigms

66 Summary of the SKA Square Kilometre Array: the next generation radio telescope: Leverages advances in computing to enable an order of magnitude leap in capabilities Mass manufacturing of many low-cost receptors Key science drivers: gravitational waves, ionisation of primordial gas by first galaxies Timeline for phase 1 (SKA1): Currently in design phase design complete 2016 Construction begins 2017 Commissioning 2019 Full operations 2022 Computing a key part of design, construction and operational phases Power budget for computing one of the major operational costs of the observatory! Design is being done by SKA SDP consortium lead by the University of Cambridge