Fujitsu Forum Human Centric Innovation. 19th 20th November. 0 Copyright 2014 FUJITSU

Transcription

1 Fujitsu Forum th 20th November Human Centric Innovation 0 Copyright 2014 FUJITSU

2 Informed Decisions Based on Climate Trend Analysis Gernot Fels Global Services & Solutions Marketing, Fujitsu Dr. Fritz Schinkel Manager for Cloud Infrastructures and Big Data Innovations, Fujitsu 1 Copyright 2014 FUJITSU

3 The Promise of Big Data Discover hidden secrets Predict opportunities Identify and minimize unknown risks Take better and faster decisions Accelerate business processes Increase performance and productivity Improve efficiency and effectiveness Profitability and competitive advantage Better utilize our planet s resources A convincing value proposition. 2 Copyright 2014 FUJITSU

4 Big Data matters to every industry Retail Finance Manufacturing Big Data Maintenance Healthcare Transportation Energy Agriculture Public Sector New opportunities, new values for enterprises and society. 3 Copyright 2014 FUJITSU

5 Weather and climate trend predictions Who needs to know credible long-term weather and climate trends? Renewable power generation (solar, wind) Power plant operation Agricultural planning (flood, pest control) Ski resort planning Communities, counties, government Transport, air-traffic control, shipping, sailing Insurance Manufacturing, retail, services TV channels Historical weather data required for trend analysis. 4 Copyright 2014 FUJITSU

6 Historical weather data for trend analysis European Centre for Medium-Range Weather Forecasts (ECMWF) Analysis of weather development Global weather data since 1979 Time series of weather maps Usable for climate research and local trend analysis Weather model ERA Interim ECMWF Re-Analysis; Interim (highest resolution) Model resolution Time interval 6h Measurement time: 0:00, 6:00, 12:00, 18:00 GMT Grid of 0,25 (4 grid points per degree) 128 meteorological indicators Time series of weather maps 5 Copyright 2014 FUJITSU

7 Meteorological indicators 10 metre U wind component Large-scale snowfall Surface net thermal radiation Vertical integral of eastward cloud liquid water flux 10 metre V wind component Logarithm of surface roughness length for heat Surface net thermal radiation, clear sky Vertical integral of eastward geopotential flux 10 metre wind gust since previous post-processing Low cloud cover Surface pressure Vertical integral of eastward heat flux 2 metre dewpoint temperature Maximum temperature at 2 metres since previous post-processing Surface roughness Vertical integral of eastward kinetic energy flux 2 metre temperature Mean sea level pressure Surface sensible heat flux Vertical integral of eastward mass flux Albedo Mean wave direction Surface solar radiation downwards Vertical integral of eastward ozone flux Boundary layer dissipation Mean wave period Surface thermal radiation downwards Vertical integral of eastward total energy flux Boundary layer height Medium cloud cover Temperature of snow layer Vertical integral of eastward water vapour flux Charnock Minimum temperature at 2 metres since previous post-processing TOA incident solar radiation Vertical integral of energy conversion Clear sky surface photosynthetically active radiation Northward gravity wave surface stress Top net solar radiation Vertical integral of kinetic energy Convective available potential energy Northward turbulent surface stress Top net solar radiation, clear sky Vertical integral of mass of atmosphere Convective precipitation Photosynthetically active radiation at the surface Top net thermal radiation Vertical integral of mass tendency Convective snowfall Runoff Top net thermal radiation, clear sky Vertical integral of northward cloud frozen water flux Downward UV radiation at the surface Sea surface temperature Total cloud cover Vertical integral of northward cloud liquid water flux Eastward gravity wave surface stress Sea-ice cover Total column ice water Vertical integral of northward geopotential flux Eastward turbulent surface stress Significant height of combined wind waves and swell Total column liquid water Vertical integral of northward heat flux Evaporation Skin reservoir content Total column ozone Vertical integral of northward kinetic energy flux Forecast albedo Skin temperature Total column water Vertical integral of northward mass flux Forecast logarithm of surface roughness for heat Snow albedo Total column water vapour Vertical integral of northward ozone flux Forecast surface roughness Snow density Total precipitation Vertical integral of northward total energy flux Gravity wave dissipation Snow depth Vertical integral of cloud frozen water Vertical integral of northward water vapour flux High cloud cover Snow evaporation Vertical integral of cloud liquid water Vertical integral of ozone Ice temperature layer 1 Snowfall Vertical integral of divergence of cloud frozen water flux Vertical integral of potential+internal energy Ice temperature layer 2 Snowmelt Vertical integral of divergence of cloud liquid water flux Vertical integral of potential+internal+latent energy Ice temperature layer 3 Soil temperature level 1 Vertical integral of divergence of geopotential flux Vertical integral of temperature Ice temperature layer 4 Soil temperature level 2 Vertical integral of divergence of kinetic energy flux Vertical integral of thermal energy Instantaneous eastward turbulent surface stress Soil temperature level 3 Vertical integral of divergence of mass flux Vertical integral of total energy Instantaneous moisture flux Soil temperature level 4 Vertical integral of divergence of moisture flux Vertical integral of water vapour Instantaneous northward turbulent surface stress Sunshine duration Vertical integral of divergence of ozone flux Volumetric soil water layer 1 Instantaneous surface sensible heat flux Surface latent heat flux Vertical integral of divergence of thermal energy flux Volumetric soil water layer 2 Large-scale precipitation Surface net solar radiation Vertical integral of divergence of total energy flux Volumetric soil water layer 3 Large-scale precipitation fraction Surface net solar radiation, clear sky Vertical integral of eastward cloud frozen water flux Volumetric soil water layer 4 6 Copyright 2014 FUJITSU

8 GRIB for historical and forecast weather data GRIdded Binary Compressed binary format Standard defined by WMO (World Meteorological Organization) Used to store weather data Based on rectangular grid Geographic coordinates as grid points 7 Copyright 2014 FUJITSU

9 Challenges and objectives Challenges Time series of global weather maps do not give immediate insight for certain location Difficult and long-lasting evaluation (e.g. for wind probability estimation) Late decisions Objectives Retrieve time series for certain location Create puncture for relevant grid points over relevant period of time Data transformation needed How many files and which data volumes? 8 Copyright 2014 FUJITSU

10 File quantities and data volumes 1 GRIB file per snapshot 4 snapshots per day 51,100 input files (timerelated) over 35 years 360 degrees of longitude 180 degrees of latitude 4 grid points per degree of longitude and latitude Add 1 grid point (north and south pole) 360 x 4 x (180 x 4 +1) = 1,038,240 grid points = output files (location-related) 128 meteorological indicators, 4 bytes each 25 TB of historical weather data Which solution concept will help? 9 Copyright 2014 FUJITSU

11 Client Distributed Parallel Processing Concept Slaves DFS Distribute data and I/O to server cluster nodes Local server storage Move computing to where data resides Shared nothing architecture Data replication to several nodes Master JobTracker TaskTracker DataNode TaskTracker Benefits NameNode DataNode High performance, fast results Unlimited scalability Fault-tolerance Cost-effective (standard servers with OSS) TaskTracker DataNode De-facto standard 10 Copyright 2014 FUJITSU

12 platform meets the challenges ECMWF data as input Load ECMWF data into HDFS Use MR to invert data From time series of world-wide weather maps Into grid point based time series of weather data Arrange as (sorted) KV pairs Key = grid point and time combined Value = Meteorological data Java apps Retrieve proximate time series, determine local weather development Visualize results Incremental update by short MR jobs Import weather history Retrieve proximate time series, determine local weather development Visualize result Invert time series 11 Copyright 2014 FUJITSU

13 Example: Wind park planning Time period: 14 years ( ) 4 snapshots per day 20,456 input files (timerelated) from ECMWF 21,221,625,600 records 1,038,240 grid points = output files (locationrelated) 12 Copyright 2014 FUJITSU

14 Solution approach effects time to value PERL ~100 min for processing data from 1 month 12 x 14 x 100 min ~ 12 days (over 14 years) 12 x 35 x 100 min ~ 30 days (over 35 years) Not acceptable in practice 13 Copyright 2014 FUJITSU

15 Solution approach effects time to value PERL ~100 min for processing data from 1 month 12 x 14 x 100 min ~ 12 days (over 14 years) 12 x 35 x 100 min ~ 30 days (over 35 years) Not acceptable in practice MapReduce 30 min for import to HDFS 141 min processing time Read HDFS files (historical data) Data transformation Write results to HDFS files ~120 x faster than script approach 8 Slave Nodes (2-socket, 6C/12T) Servers not fully utilized Potential for improvement by removing other workload Speed advantage by parallelization. 14 Copyright 2014 FUJITSU

16 Solution approach effects time to value PERL ~100 min for processing data from 1 month 12 x 14 x 100 min ~ 12 days (over 14 years) 12 x 35 x 100 min ~ 30 days (over 35 years) Not acceptable in practice MapReduce 30 min for import to HDFS 141 min processing time Read HDFS files (historical data) Data transformation Write results to HDFS files ~120 x faster than script approach 8 Slave Nodes (2-socket, 6C/12T) Servers not fully utilized Potential for improvement by removing other workload Options for further acceleration? 15 Copyright 2014 FUJITSU

17 In-memory platform as option For single users Hadoop platform is sufficient Retrieval and visualization within seconds Increasing response times Increasing number of users Increasing number of queries Complex queries, e.g. where in certain geographic area are most favorite locations for certain plans Solution: IMDG Accelerate retrieval and visualization Pre-defined queries memory-resident Import weather history Retrieve proximate time series, determine local weather development Visualize result Invert time series In-memory platforms help cope with any level of complexity. 16 Copyright 2014 FUJITSU

18 Data transformation Just 1x or more often? Depending on use case no one-time act New inversion of weather maps with new questions Deduction from meteorological indicators in global weather maps Example: Max. wind speed Peak speeds of crossing weather front occur only shortly at one location Often fail at 6 hrs grid Determine front on weather map timely before and thereafter What is the effort to realize new questions? 17 Copyright 2014 FUJITSU

19 Dreamlike Big Data Display weather data as table Spreadsheet like Excel Apply meteorological formulas directly to sample data Check partial results at once in spreadsheet Fast test run on significant (filtered) test data set Simple expansion to total data set and visualization Process large data volumes, but avoid programming MR jobs? Big Data for business users. 18 Copyright 2014 FUJITSU

20 Dreamlike Big Data Display weather data as table Spreadsheet like Excel Apply meteorological formulas directly to sample data Check partial results at once in spreadsheet Fast test run on significant (filtered) test data set Simple expansion to total data set and visualization Big Data for business users. 19 Copyright 2014 FUJITSU

21 Is this a Big Data project? Volume Variety Versatility Velocity Value 1 data source Structured data Data is not generated at high speed Analysis not always time-critical 25 TB x 2 is a considerable volume Traditional technologies do not help Big Data technologies solve customer problem Affordable Scalable with growth Expected processing time can be controlled Indeed no day-to-day Big Data project, but a very interesting one. 20 Copyright 2014 FUJITSU

22 Data in motion Data at rest Big Data infrastructure Data Sources Analytics Platform Access IMDB DB / DW IMDG Distributed Parallel Processing NoSQL IMDB NoSQL Apps Services Queries. Un- / Semi-/ Polystructured data CEP IMDG FS DB / DW IMDG Visualization Reporting Notification Various data Consolidated data Distilled essence Applied knowledge Extract, Collect Clean, Transform Analyze, Visualize Decide, Act 21 Copyright 2014 FUJITSU

23 Data in motion Data at rest Big Data infrastructure Data Sources Analytics Platform Access IMDB IMDB Apps Services Queries. DB / DW IMDG NoSQL NoSQL Un- / Semi-/ Polystructured data CEP IMDG FS DB / DW IMDG Visualization Reporting Notification Various data Consolidated data Distilled essence Applied knowledge Extract, Collect Clean, Transform Analyze, Visualize Decide, Act 22 Copyright 2014 FUJITSU

24 Data in motion Data at rest Big Data infrastructure Data Sources Analytics Platform Access IMDB IMDB Apps Services Queries. DB / DW IMDG NoSQL NoSQL Un- / Semi-/ Polystructured data CEP IMDG FS DB / DW Visualization Reporting Notification Various data Consolidated data Distilled essence Applied knowledge Extract, Collect Clean, Transform Analyze, Visualize Decide, Act 23 Copyright 2014 FUJITSU

25 Data in motion Data at rest Big Data infrastructure Data Sources Analytics Platform Access IMDB IMDB Apps Services Queries. DB / DW IMDG NoSQL NoSQL Un- / Semi-/ Polystructured data CEP IMDG FS DB / DW Visualization Reporting Notification Various data Consolidated data Distilled essence Applied knowledge Extract, Collect Clean, Transform Analyze, Visualize Decide, Act 24 Copyright 2014 FUJITSU

26 How can help Complete analytics platform Infrastructure and services Consulting, introduction, operation, maintenance Apps for analysis and visualization Integrated Systems for fast deployment Location-based time series as cloud service Weather prediction expertise Everything from a single source: Simple, fast, without risk. 25 Copyright 2014 FUJITSU

27 FUJITSU Showcase 26 Copyright 2014 FUJITSU

28 Fujitsu showcase Questions to be answered What are the wind trends in a certain area? Is weather better during weekends or on working days? In which areas which differences? 27 Copyright 2014 FUJITSU

29 Summary 28 Copyright 2014 FUJITSU

30 Summary Big Data one of today s megatrends Promising value proposition Exciting use cases across industries Knowledge about future weather and climate is valuable for many target groups Historical data available Transformation needed to get desired insight and recognize trends End-to-end solutions from Fujitsu Integrated systems for fast-time to production Supplementing services You d like to look into the future? Have a word with Fujitsu. 29 Copyright 2014 FUJITSU

31 Thank you for listening

32 Appendix 31 Copyright 2014 FUJITSU

33 Fujitsu Showcase: Wind Trends (1) Select location in map or satellite view by click or select previously saved location 32 Copyright 2014 FUJITSU

34 Fujitsu Showcase: Wind Trends (2) Wait a second 33 Copyright 2014 FUJITSU

35 Fujitsu Showcase: Wind Trends (3) See the wind, temperature and air pressure Move over the charts and see individual values 34 Copyright 2014 FUJITSU

36 Fujitsu Showcase: Wind Trends (4) Select zoom windows to see details in wind speed, temperature and air pressure 35 Copyright 2014 FUJITSU

37 Fujitsu Showcase: Wind Trends (5) Distribution of wind speed over time, and distribution of wind frequency and speed along wind direction is displayed and 36 Copyright 2014 FUJITSU

38 Fujitsu Showcase: Wind Trends (6) Distribution of wind speed over time, and distribution of wind frequency and speed along wind direction is displayed and month to be taken into account can be restricted and animated 37 Copyright 2014 FUJITSU

39 Fujitsu Showcase: Wind Trends (7) Select year or span of years for long term trends 38 Copyright 2014 FUJITSU

40 Fujitsu Showcase: Wind Trends (8) Munich Mecklenburg- Vorpommern Somalia Paderborn Borkum (off-shore) Cape Horn 39 Copyright 2014 FUJITSU

41 Fujitsu Showcase: Weekdays and Weather (1) Get complete page automatically published Number of grid points with maximum / minimum temperature on certain weekday Locations with most significant span between warmest and coldest weekday average as map and as list Locations with most significant span between warmest and coldest weekday average and warmest day on a certain weekday 40 Copyright 2014 FUJITSU

42 Fujitsu Showcase: Weekdays and Weather (2) Visualization GUI to study the span of weekday mean temperature at certain places and to look for possible reasons Map colored for high span of weekday mean temperature 41 Copyright 2014 FUJITSU

43 Fujitsu Showcase: Weekdays and Weather (3) Sliders for span threshold, contrast and opacity of coloring 42 Copyright 2014 FUJITSU

44 Fujitsu Showcase: Weekdays and Weather (4) And an adjustment for grid points with low temperature span over the complete observation time 43 Copyright 2014 FUJITSU

45 Fujitsu Showcase: Weekdays and Weather (5) Using the color settings and the zooming into the map we can find areas with significant differences of weekday mean values in the observed timeframe 44 Copyright 2014 FUJITSU

46 Fujitsu Showcase: Weekdays and Weather (6) Click to a certain position shows the curve of average temperature for the weekdays, the coordinates and the total min/max temperature of the point 45 Copyright 2014 FUJITSU

47 Fujitsu Showcase: Weekdays and Weather (7) Map and satellite can be used to find possible reasons for mean temperature related to weekdays 46 Copyright 2014 FUJITSU

48 Fujitsu Showcase: Weekdays and Weather (8) Zoom into the source of the color cloud Industrial complex is shut down on Sunday? 47 Copyright 2014 FUJITSU

49 Fujitsu Showcase: Weekdays and Weather (9) US east cost is cooler on Sunday / Monday Is traffic system heating the atmosphere over the week? 48 Copyright 2014 FUJITSU

50 Fujitsu Showcase: Weekdays and Weather (10) South of Hudson Bay is an area with Wednesday mean temperature approx. 1C higher than on Saturday Does wood industry influence the temperature in the rhythm of the week? 49 Copyright 2014 FUJITSU

51 50 Copyright 2014 FUJITSU