Southeast University and AMS-02 Experiment

Transcription

1 The 14th International Conference on Cryptology and Network Security (CANS 2015) Southeast University and AMS-02 Experiment Fang Dong School of Computer Science and Engineering, Southeast University

2 Outline n Background of AMS n SEU s contribution to AMS n Conclusion and future work 2

3 The Origin of Dark Matter ~ 90% of Matter in the Universe is not visible and is called Dark Matter A Galaxy as seen by telescope If we could see Dark Matter in the Galaxy

4 Help scientists find the source of the dark matter and measuring cosmic ray sources as well as antimatter universe Alpha Magnetic Spectrometer AMS is the only large-scale physics experiment that 4 directly searches for dark matter in the space

5 AMS is US Dept of Energy (DOE) led International Collaboration 16 Countries and regions, 60 Institutes and 600 Physicists, 17 years USA A&M FLORIDA UNIV. JOHNS HOPKINS UNIV. MIT - CAMBRIDGE NASA GODDARD SPACE FLIGHT CENTER NASA JOHNSON SPACE CENTER UNIV. OF MARYLAND-DEPRT OF PHYSICS UNIV. OF MARYLAND-E.W.S. S.CENTER YALE UNIV. - NEW HAVEN MEXICO UNAM DENMARK UNIV. OF AARHUS FINLAND HELSINKI UNIV. UNIV. OF TURKU FRANCE GAM MONTPELLIER LAPP ANNECY LPSC GRENOBLE GERMANY RWTH-I RWTH-III MAX-PLANK INST. UNIV. OF KARLSRUHE ITALY ASI CARSO TRIESTE IROE FLORENCE INFN & UNIV. OF BOLOGNA INFN & UNIV. OF MILANO INFN & UNIV. OF PERUGIA INFN & UNIV. OF PISA INFN & UNIV. OF ROMA INFN & UNIV. OF SIENA NETHERLANDS ESA-ESTEC NIKHEF NLR PORTUGAL LAB. OF INSTRUM. LISBON ROMANIA ISS UNIV. OF BUCHAREST RUSSIA I.K.I. ITEP KURCHATOV INST. MOSCOW STATE UNIV. SPAIN CIEMAT - MADRID I.A.C. CANARIAS. SWITZERLAND ETH-ZURICH UNIV. OF GENEVA 中国 中科院电 工所中科院 高能所东南 大学上海交 大中 山 大学 山东 大学航天部 一院航天部五院 中 山科学院中央研究院中央 大学成功 大学太空计划室交通 大学 KOREA EWHA KYUNGPOOK NAT.UNIV.

6 6 5m x 4m x 3m 7.5 tons TRD TOF 1, 2 300,000 electronic channels 650 microprocessors Silicon layer Magnet 7 Silicon layers Radiators TOF RICH ECAL 11,000 Photo Sensors Silicon layer 6

7 STS-134 launch May 16, 08:56 AM

8 May 19: AMS installation completed at 5:15 CDT, start taking data 9:35 CDT

9 AMS Data Flow Tracking and Data Relay Satellite,TDRS KU band 太空粒 子束 Huntsville, AL Marshall Space Flight Center AMS Scientific Operation Center JSC CERN POCC POCC Ø AMS Flight Status Monitoring Ø AMS Flight Status Monitoring SOC Ø Ø Ø Ø Data reconstruction Data analysis Data distribution Monte Carlo production SEU RWTH Milano CIEMAT Regional SOC Ø Ø Ø Ø Data storage Data reconstruction Monte Carlo production Data analysis 9

10 Missions of AMS-SOC q Data storage data reconstruction Monte Carlo simulation physics analysis massive data transfer M C Sim ulatio n D etecto r Sim ulatio n Sim ulatio n D ata D ata R eco nstructio n GEANT AMS-02 Package Physics A nalysis R esults D A Q R aw D ata D ata R eco nstructio n ROOT V isulizatio n C alib ratio n 10

11 Outline n Background of AMS n SEU s contribution to AMS n Conclusion and future work 11

12 AMS 1. AMS-C 2. AMS-AIS SEU USA MIT CAMBRIDGE - NASA GODDARD SPACE FLIGHT CENTER NASA JOHNSON SPACE CENTER NASA KENNEDY SPACE CENTER FLORIDA STATE UNIVERSITY (NAT. HIGH MAG. FIELD LAB) TEXAS A&M UNIVERSITY UNIV. OF MARYLAND YALE UNIV. NEW HAVEN MEXICO UNAM - JSC NETHERLANDS ESA - ESTEC NIKHEF NLR FRANCE GAM MONTPELLIER LAPP ANNECY LPSC GRENOBLE SPAIN CIEMAT - MADRID I.A.C. CANARIAS. PORTUGAL LAB. OF INSTRUM. LISBON DENMARK UNIV. OF AARHUS CERN FINLAND HELSINKI UNIV. UNIV. OF TURKU GERMANY RWTH - I RWTH - III MAX - PLANK INST. UNIV. OF KARLSRUHE SWITZERLAND ETH - ZURICH UNIV. OF GENEVA ITALY ASI CARSO TRIESTE IROE FLORENCE INFN & UNIV. OF BOLOGNA INFN & UNIV. OF MILANO INFN & UNIV. OF PERUGIA INFN & UNIV. OF PISA INFN & UNIV. OF ROMA INFN & UNIV. OF SIENA ROMANIA ISS, UNIV. OF BUCHAREST RUSSIA I.K.I. ITEP KURCHATOV INST. MOSCOW STATE UNIV. KOREA EWHA KYUNGPOOK NAT.UNIV. CHINA NLAA (Beijing) BISEE (Beijing) IEE (Beijing) IHEP (Beijing) SJTU (Shanghai) SEU (Nanjing) SYSU (Guangzhou) SDU (Jinan) SEU SINICA (Taiwan) AIDC (Taiwan) CSIST(Taiwan) NCU (Chung Li) NCKU (Tainan) NCTU(Hsinchu) NSPO(Hsinchu) AMS-C@SEU AMS-SOC@SEU Starting its collaboration with AMS from 2002, SEU is the first Chinese university to join in AMS experiment. 12

13 Talents exchange between SEU and AMS SEU s Collaboration with AMS and MIT lasts for more than a decade ~40 SEU Professors/Assistant Professors/students visited MIT MIT faculties including Nobel laureate Professor Samuel Ting Research Professor Vitaly Choutko visited SEU regularly in the past few years Prof. Samuel Ting visited SEU 13

14 Talents exchange between SEU and AMS Prof. Li Qi at MIT Prof. Luo Junzhou at NASA Prof. Vitaly Choutko at SEU Discussions at CERN 14

15 High-tech breakthroughs at SEU AMS-02 silicon trackers AMS-02 Control System AMS-C at SEU Prof. Samuel Ting at SEU 15

16 AMS 1. AMS-C 2. AMS-AIS 3. AMS-SOC SEU USA MIT CAMBRIDGE - NASA GODDARD SPACE FLIGHT CENTER NASA JOHNSON SPACE CENTER NASA KENNEDY SPACE CENTER FLORIDA STATE UNIVERSITY (NAT. HIGH MAG. FIELD LAB) TEXAS A&M UNIVERSITY UNIV. OF MARYLAND YALE UNIV. NEW HAVEN MEXICO UNAM - JSC NETHERLANDS ESA - ESTEC NIKHEF NLR FRANCE GAM MONTPELLIER LAPP ANNECY LPSC GRENOBLE SPAIN CIEMAT - MADRID I.A.C. CANARIAS. PORTUGAL LAB. OF INSTRUM. LISBON DENMARK UNIV. OF AARHUS CERN FINLAND HELSINKI UNIV. UNIV. OF TURKU GERMANY RWTH - I RWTH - III MAX - PLANK INST. UNIV. OF KARLSRUHE SWITZERLAND ETH - ZURICH UNIV. OF GENEVA ITALY ASI CARSO TRIESTE IROE FLORENCE INFN & UNIV. OF BOLOGNA INFN & UNIV. OF MILANO INFN & UNIV. OF PERUGIA INFN & UNIV. OF PISA INFN & UNIV. OF ROMA INFN & UNIV. OF SIENA ROMANIA ISS, UNIV. OF BUCHAREST RUSSIA I.K.I. ITEP KURCHATOV INST. MOSCOW STATE UNIV. KOREA EWHA KYUNGPOOK NAT.UNIV. CHINA NLAA (Beijing) BISEE (Beijing) IEE (Beijing) IHEP (Beijing) SJTU (Shanghai) SEU (Nanjing) SYSU (Guangzhou) SDU (Jinan) SEU SINICA (Taiwan) AIDC (Taiwan) CSIST(Taiwan) NCU (Chung Li) NCKU (Tainan) NCTU(Hsinchu) NSPO(Hsinchu) AMS-C@SEU AMS-SOC@SEU Starting its collaboration with AMS from 2002, SEU is the first Chinese university to join in AMS experiment. 16

17 AMS-02 SOC at SEU q SEU data center dedicated to AMS experiment q 5500 CPU cores (70 TFLOPS at peak) and 920TB storage. 40Gbps Infiniband 1Gbps Ethernet To CERNET Beijing Exchange Point 4 IBM 42U Rack To SEU Campus Network 40Gbps QDR Infiniband Switch IBM X3550 M3 8Gbps Fiber Channel IBM X3650 M3 8 IBM 42U Rack 1Gbps Dedicated Link FC SAN Switch IBM X3850 X5 1Gpbs Ethernet Switch q Docker OpenStack Hadoop IBM H22 Blade Server q 1Gb/s Docker dedicated network between SEU and CERNET Beijing gateway, 2.5Gb between CERNET and CERN IBM DS5300 q Cloud based AMS-SOC for better supporting AMS data storage computing and data transfer 17

18 SEU s AMS Data Transfer Network Topology of China Education and Research NETwork(CERNET) Backbone CERN NLAA Beijing CNGI-6IX Wuhan 1GPOS Dedicated for AMS 10GPOS 2.5GPOS IPv6 Link Nanjing SEU Shanghai Tier 1 Tier 2 Guangzhou 18

19 A Glance at SEU s AMS-SOC (cost: RMB 25M) 19

20 Prof. Ting and the MIT delegation visited SEU s AMS-SOC to discuss the data analysis solutions 20

21 BladeCenter BladeCenter BladeCenter BladeCenter BladeCenter BladeCenter BladeCenter BladeCenter Data processing at SEU s AMS SOC Facility for AMS MC production:hpc Cluster(168 physical servers) and Facility cloud for based AMS VM data Cluster(more reconstruction than180 :HPC instances) Cluster(168 physical servers), 1Gb/s network Physics between analysis SEU and CERN M C Pro d uctio n HPC Cluster (Physical nodes) D etecto r Sim ulatio n AMS user Login 2 GEANT 1 Sim ulatio n Job D ata Data submission D ata R eco nstructio analysis n Login node c01 c02 c12 c16 c17 6. production AMS User AMS reconstruction jobs jobs v5.00mcscripts.tar.gz 7. status update v5.00mcdb.tar.gz v5.00mcdb.addon.tar.gz AMS-02 Job 6 Remotely transfer data Package scheduling 7. status update C alib ratio n Cluster (VMs) Ser OS 5. Job submit Ser OS Physics A nalysis ROOT/Geant/ MC Job request 3. Job/Env. registration v5.00mcscripts.tar.gz v5.00mcdb.tar.gz SEU s AMS SOC v5.00mcdb.addon.tar.gz 8. data transfer Ser OS Ser OS c18 Filemonitor &Database R esults CERN REC jobs downlad 4. Env. update copy REC jobs Remotely transfer jobs D A Q 3 D ata R aw D ata 4 Computing ROOT 52. MC Software Job/Env. download V isualizatio n R eco nstructio resources n resources 168 IBM LSF Server ams@cern.ch ams01@cern.ch 42 IBM IBM Cluster@SEU Database Sugon Cluster@SEU HS22 blade servers LSF Flex240 blade servers v5.00mcscripts.tar.gz v5.00mcdb.tar.gz v5.00mcdb.addon.tar.gz Analysis results Data Storage Data Transfer Servers 21

22 Data processing at SEU s AMS SOC Data Production at SEU CPU hours spent at SEU SEU q 696TB data are produced at SEU, including 138TB Raw data,368tb Monte Carlo data and 190TB Reconstructed data q 1.53M CPU hours are spent at SEU: 1 st place in 6 regional AMS SOC Data source: 22

23 Data transfer at SEU s AMS SOC Data transfer architecture CERNè SEU( Raw Data ) client:lftp server:vsftp Request: 200GB/day Base server at CERN Upload *.* CERN SEU Client (LFTP) Server (VSFTP) 23

24 Data transfer at SEU s AMS SOC Data transfer architecture SEUè CERN( MC Simulation Data / Reconstructed Data ) client:lftp server:vsftp Request: 1~2TB/day client (LFTP) Server (VSFTP) *.* Base server at CERN download *.* CERN SEU *.*

25 Data transfer at SEU s AMS SOC Data transfer improvement by optimizing two parameters: the size of socket buffer at both sending and receiving sides; the number of parallel connections Increasing the socket buffer size of the sending and receiving side will help to improve the data transfer performance Increasing the number of parallel connections does not always increase the transfer performance Transfer bandwidth with different number of parallel connections Transfer time and packet loss rate with different number of parallel connections 25

26 Data transfer at SEU s AMS SOC Data Transfer Status AMS data transfer reaches 400Mbit/sec after optimization. 26

27 Data transfer at SEU s AMS SOC 1.4 Daily transferred data size (TB) May Jun Jul Aug Sep Date SEU is able to transfer more than 1 TB AMS data to CERN per day 27

28 Researches on AMS big data processing AMS application layer MC Simulation Data Analysis Data processing layer Graph Data Query Processing Data Reconstruction Streaming data processing Big data storage and management layer Pre-processing Placement Migration Cloud supporting platform layer Virtualization Elastic Resource Management C o m p uting R eso urces Sto rag e R eso urces Hardwares in DC N etw o rk R eso urces 28

29 Cloud based AMS big data processing platform AMS users AMS-SOC portal SEU users Raw data Task Scheduler AMS applications MC Simulation Data Reconstruction Data Analysis Map/Reduce Big data processing Data Query Data Analysis Streaming Data Processing Big data processing platforms Hadoop Yarn/Spark/Tez/Storm Data Storage Data Placement Data Migration Elastic Resource Management Ser OS Openstack Ser OS o Ceph SSD SCSI/ SATA TAPE Hardwares in DC 29

30 Outline n Background of AMS n SEU s contribution to AMS n Conclusion and future work 30

31 Conclusion q After these 13 years collaborations, we have obtained some fruitful achievements Ø Successfully construct AMS-C, AMS-AIS and AMS-SOC at SEU Ø Process and transfer more than 700TB scientific data Ø Breakthrough several key technologies Ø Exchange more than 40 teachers and students 31

32 Future Work q AMS-02 will continuously produce massive data(~4pb in next 10 years of AMS operation), including : Ø raw data (~40TB/year) Ø reconstructed data(~200tb/year) Ø MC data(~200tb/year) q Big data processing techniques need to be further employed including: Ø Geo-distributed data analysis among multiple DCs Ø High performance data transfer between AMS-SOCs 32

33 Thank You!