Dynamic Data-Driven Adaptive Sampling and Monitoring of Big Spatial-Temporal Data Streams for Real-Time Solar Flare Detection

Size: px

Start display at page:

Download "Dynamic Data-Driven Adaptive Sampling and Monitoring of Big Spatial-Temporal Data Streams for Real-Time Solar Flare Detection"

Diane Jennings
5 years ago
Views:

Engineering University of Wisconsin-Madison

1 Dynamic Data-Driven Adaptive Sampling and Monitoring of Big Spatial-Temporal Data Streams for Real-Time Solar Flare Detection Dr. Kaibo Liu Department of Industrial and Systems Engineering University of Wisconsin-Madison Date: 8/10/2016 Lab for System Informatics and Data Analytics (SIDA) 1

2 Outline Motivation State of the art Proposed DDDAS framework Data-Driven Dynamic Sampling Strategy Case study Conclusion 2

This provides an unprecedented opportunity to gain systemwide situational awareness through real-time anomaly detection and fault localization.

3 Motivation & Applications With the advancement of sensing technique and data collection capability, Big Data Streams have become widely available in many DoD applications. This provides an unprecedented opportunity to gain systemwide situational awareness through real-time anomaly detection and fault localization. The emerging NASA Solar Dynamics Observatory (SDO) continuously monitors the dynamic solar activities for 24 hours/7 days a week dimensional observations generate an image every 0.75 second produce 1.5 TB big data per day Solar flare detection 3 Source: NASA

4 Motivation & Applications The solar flare activities have a close relationship with Air Forces equipment and applications. A solar flare is a sudden, transient, and intense variation in brightness: significantly affect Earth s ionosphere, causing hours-long disruptions in radio communications Affect GPS receivers and satellites, making it very difficult for search and rescue in a war zone lead to failures in large-scale power-grid with cascading effects Real-time detection system for the solar flare by exploiting the Big Data Streams of solar images is highly desired. 4

5 Challenges Big Data Streams place critical requirements and resources constraints for data communication and processing in real time Send only 6 images back every minute for real-time analysis given transmissions rate 130 million bits/second The occurrence of solar flare is naturally complicated (depends on the cycle and the inherent dynamics and randomness of solar activities) sparse (with a small signal-to-noise ratio (SNR)) transient (only lasts for minutes and hard to predict) Currently, there is a lack of efficient online monitoring scheme tailored to these unique characteristics. 5

6 State of the art Existing approaches to process monitoring focus on fixed sub-region (rigid spatial domain) assume that the locations of anomaly event are known fail to capture the dynamic features of solar flare events sample whole image at fixed frequency (rigid temporal domain) result in a large detection delay or miss the event Adaptive sampling strategy: require large amount of historical information 6

7 Problem Formulation & Goal m physical variables M = 1,, m and q (q m) sampling resources in a system. When no solar flare occurs: Each variable k ~ N(0,1) At some unknown time ν: Anomaly occurs at certain variables and will affect an unknown subset of data streams Each variable k ~ N(u k, 1) Goal: Based on dynamic observations in real time, actively decide which dataset to observe at the next time for quick detection of anomaly event while still maintaining a system-wide false alarm rate. 7

Case Study Solar Flare Detection Assume only 2000 out of 67744

8 Case Study Solar Flare Detection Assume only 2000 out of pixels are available Accounts for only 2.95% partial information 8

9 Case Study Solar Flare Detection Assume only 2000 out of pixels are available Accounts for only 2.95% partial information 9

10 Case Study Solar Flare Detection Assume only 2000 out of pixels are available Accounts for only 2.95% partial information 10

11 Case Study Solar Flare Detection Assume only 2000 out of pixels are available Accounts for only 2.95% partial information 11

12 Result Comparison 12 Our method base on 2000 pixels Generalized likelihood ratio test approach based on pixels First solar flare t = 90 t = 91 Second solar flare t = 121 t = 117 Algorithm type Real-time Monitoring Efficient (dynamic, recursive) Yes Inefficient (non-dynamic, non-recursive) No

13 Summary of the proposed sampling strategy A systematic adaptive sampling strategy is proposed for real-time monitoring of Big Data streams with dynamically selected partial information. Scalability: linear in the number of data streams Adaptability: Quickly detect a wide range of possible changes with no prior knowledge of the potential anomaly events by adaptively adjusting to the event locations; Actively select the data streams to observe from the whole streaming data to maximize the sensitivity for anomaly detection with consideration of resource constraints. 13

14 Conclusion and Impact to Air Force It is critically important for the Air Force to make rapid decisions in a battlefield based on real-time Big Data continuously collected from massive sensor. The proposed method is generic and can be applied to a wide range of applications, such as intrusion detection, unmanned vehicle surveillance, cybersecurity Early detection and localization of these anomaly events will enhance system-wide situational awareness to support warfighters/military operations, prevent damages, reduce cost, improve efficiency, and save billions of lives. 14

15 Thank you for coming! Questions? 15

Collaborative Target Detection in Wireless Sensor Networks with Reactive Mobility

1 / 24 Collaborative Target Detection in Wireless Sensor Networks with Reactive Mobility Rui Tan 1 Guoliang Xing 1 Jianping Wang 1 Hing Cheung So 2 1 Department of Computer Science City University of Hong