Real-Time Big Data Analytical Architecture for Remoting Sensing Application

Transcription

1 Real-Time Big Data Analytical Architecture for Remoting Sensing Application Author: M. M. U. Rathore, A. Paul, A. Ahmad, B.-W. Chen, B. Hunag and W. Ji Presenter: Junyi Shen

2 Big Data High speed continuous stream of data or high volume offline data

3 Why? Various fields of remote sensory satellite image data are needed Consequences of transformation of remotely sensed data to the scientific understanding are critical Sensors produce an overwhelming amount of raw data Data collected from remote area are not in a format ready for analysis

4 Architecture 1.jpg DADU: Compilation, storage of results, and generation of decision DPU: Filtration, load balancing, and parallel processing RSDU: Data acquisition from satellite and delivery to Base Station

5 Remote Sensing Big Data Acquisition Unit (RSDU) Ground station Remove distortions from data Assume satellite can correct error data Raw data -> image format Integrate data Decrease storage cost Improve analysis accuracy

6 Data Processing Unit (DPU) Identify useful information, block and discard the rest Divide filtered data into parts and assign to various servers Each has its algorithm implementation Perform tasks independently and in parallel

7 Data Analysis and Decision Unit (DADU) Display and broadcast the decision Discover hidden things and make decisions Aggregate partial results received in DPU and organize into proper form Any server can access

8 Flowchart

9 Case Study Detect Land, Sea or Ice Area for remote sensing Big Data images

10 Data Set Take satellite-sensed Big Data samples from European Satellite Agency (ESA) to analyze land, sea and ice separately Analyze the ENVISAT mission data sets (continuously providing global measurements for the earth including sea, land and ice since 2002)

11 Analysis Ø Mean value for land areas are quite lower compared to sea area Land normally has greenery (except dessert) Color of other objects is nearer to black Ø SD values of land are higher than sea area Sea has one color Few particles on the surface of sea Land has more different colors on its surface

12 Reason: Large lake between land surface PVD for land area PVD for Sea area

13 Algorithm Design Algorithm IV Algorithm III Algorithm II Algorithm I

14 Parameters and Variables Introduction

15 Algorithm I && II

16 Algorithm III & IV

17 Results and Evaluation TP : percentage of land blocks correctly identified FP : percentage of sea blocks incorrectly identified as land blocks

18 Hadoop MapReduce vs. Simple Java

19 Reference [1] M. M. U. Rathore, A. Paul, A. Ahmad, B.-W. Chen, B. Hunag and W. Ji, "Real-Time Big Data Analytical Architecture for Remote Sensing Application", IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 8, no. 10, pp , [2] S. Kalluri, Z. Zhang, J. JaJa, S. Liang, and J. Townshend, Characterizing land surface anisotropy from AVHRR data at a global scale using high performance computing, Int. J. Remote Sens., vol. 22, pp , [3] European Space Agency. (Oct. 14, 2014). [2312] [Online]. Available: [4] EnviSat, A. S. A. R. Product Handbook, European Space Agency, Issue 2.2, Feb., 2007.

20 Discussion - Strength & Weakness A genetic architecture for any type of remote sensing real-time Big Data analysis Efficiently process and analyze real-time and offline data Detail explanation of architecture implementation in real case RSDU assumes satellite can correct error data. Does it work for all the cases? How to classify real-time Big Data and offline Big Data as it is not clearly described in this paper? A simple case study can reflect the effectiveness of this architecture?

21 Discussion - Future Work o Add erroneous data handler unit in RSDU part o Perform complex analysis on earth observatory data for real time decision making o Compatible for Big Data analysis for more applications

22 Discussion Related Papers q Md. Ali Hossain, Xiuping Jia and Jón Atli Benediktsson, "One-Class Oriented Feature Selection and Classification of Heterogeneous Remote Sensing Images", Selected Topics in Applied Earth Observations and Remote Sensing IEEE Journal of, vol. 9, pp , 2016, ISSN Propose a one-class oriented approach for effective feature selection and classification of remote sensing images. q Alexandru-Cosmin Grivei and Mihai Datcu, "HyperMINE An earth observation spatio-temporal data mining system", Telecommunications Forum Telfor (TELFOR) rd, pp , Present An Earth Observation Spatio-Temporal Data Mining System (HyperMINE), which integrates fast and complex query methods in order to generate SITS (Satellite Image Time Series) regarded as a data hypercube. The system is built on a modular multilayer architecture that allows effective processing of various sources of data.

23 Discussion - Questions Ø Is this proposed architecture essential for remote sensing Big Data? Can we use or modify the already existing architecture used for common big data such as video/audio and to deal with the remote sensing data? Ø Can we improve the efficiency of DPU by classifying and storing different information fetched from RSDU? Getting data from RSDU and discarding useless information every time cost high. Ø From simple case study in this paper, we can see analysis processing time is not low enough. So is it positive to deal with complex analysis of Big Data in real-time? Ø Hadoop implementation and simple Java implementation have difference performances when product size is increasing. Is it possible to make a combination to improve the whole efficiency?