Analysis of United States Rainfall

Size: px

Start display at page:

Download "Analysis of United States Rainfall"

Merryl Freeman
5 years ago
Views:

1 Analysis of United States Rainfall Trevyn Currie, Stephen Blatt Abstract Using hourly rainfall data in the United States, we used SQL to construct a data mart that allows for querying data based on location and time period. For the sake of not being too overwhelmed with data, we limited our data to the states of California, Oregon, Washington and Nevada. We then used clustering methods to determine which locations had similar patterns in monthly rainfall. Lastly, we graphed various clusters by longitude and latitude using different symbols for different clusters to visually identify areas of similar patterns of rainfall. Keywords - historical rainfall; extraction using SQL; rainfall tendencies; K-mean 2 Introduction Understanding weather is one of the most ancient problems our species has struggled to understand. Since the beginning of agricultural societies we have sought to understand when it will rain and when there will be a drought. We have tried to understand the reasons and patterns behind weather. Knowing when it would rain was vital to being able to feed our populations. Being able to predict storms allowed people to know when it was safe to travel. These ancient questions have been asked for a long time and still much is left unanswered. While we may not have fully answered these ancient questions, hopefully our works will allow a greater understanding of where and how much it rains, and what patterns can be recognized. We knew it was very important to find an easy way to extract rainfall from previously gathered data. We used a small segment of United States Hourly rainfall data in order to construct a database and search for patterns. 3 US Hourly Rainfall data Our data set comes from the hourly rainfall data from the National Oceanic and Atmospheric Administration (NOAA). The NOAA includes the National Centers for Environmental Information (NCEI). The NCEI is responsible for hosting and providing access to one of the most significant archives on Earth, with comprehensive oceanic, atmospheric, and geophysical data. [2] The Data set we used is hourly rainfall measurements from 5500 land based stations. The Data contains a Location, Day, Hour and amount of rainfall. In addition for each station there is a Name, State, County, Longitude, Latitude and Altitude. The data we used contains millions of records. Our data was split into two different sets of files. The first was the hourly data files. The second was the station description file. Each record in the hourly data files starts with thirty characters to describe station id and date of measurement. This was followed by a series of hourly measurements. Each measurement contains an hour and an amount of rainfall. Each record in the station information file contains a unique station id, a name, and various geographical descriptive data.

Image 1: Data Mart Star Schema 4 Using Python to Preprocess Data We used python scripts to prepare data for entry into the data mart.

We then wrote those fields into a Comma Separated Values file (CSV) (See Figure 2). 5 Rainfall Data Mart We used a simple SQL database for our Data Mart (See Image 1).

Our data mart can answer questions related to monthly totals or which stations have the most (or least rainfall) Image 2: Sample of python code used for preprocessing Our data mart

2 Image 1: Data Mart Star Schema 4 Using Python to Preprocess Data We used python scripts to prepare data for entry into the data mart. We used the powerful and simple Python string processing capacity to split up the fixed length fields in the records. We then wrote those fields into a Comma Separated Values file (CSV) (See Figure 2). 5 Rainfall Data Mart We used a simple SQL database for our Data Mart (See Image 1). We than designed queries to access the data mart via a variety of criteria. Our data mart can answer questions related to monthly totals or which stations have the most (or least rainfall) Image 2: Sample of python code used for preprocessing Our data mart includes views that show monthly total rainfall for each station. (Image 3) Some example questions our Data Mart can answer are: 1. Which Station in California had the most rainfall in 2012? (Images 4 and 5) 2. Which Stations in Nevada had the most rainfall in March 2012? (Images 6 and 7) 3. When did Grass Valley California have the most rainfall? (Images 8 and 9) Image 3: Query to create monthly summary view for January

Image 4: Question 1 Query Image 6: Question 2 Query Image 5: Question 1 Result Image 7: Question 2 Result Image 1: Question 3 Query Image 9: Question 3 Result 6

Then, we used orange to apply clustering to the resulting data set. Using K-Means clustering we found that the data was divided into two clusters.

3 Image 4: Question 1 Query Image 6: Question 2 Query Image 5: Question 1 Result Image 7: Question 2 Result Image 1: Question 3 Query Image 9: Question 3 Result 6 Clustering locations with Similar Rainfall First, we created a table that contained the monthly totals for every station. Then, we used orange to apply clustering to the resulting data set. Using K-Means clustering we found that the data was divided into two clusters. Cluster 1 (O) tend to have less rainfall while Cluster 2 (X) tends to have more rainfall (Image 10). As shown in the scatter plot. More rainfall tends to occur in Northern California, and Western Oregon and Washington. Less rainfall occurs in

4 Nevada, Southern California, and Eastern Oregon and Washington. The pattern of where the clusters appear closely matches what one would expect to see in a climate map. Various techniques have been used in the past in order to measure rainfall data. Past studies are worth taking a look at when conducting research similar to those studies. The futuristic idea of gathering rainfall data from vehicles in real time has been implemented multiple times [1]. In this study, modern cars that have rainfall sensors connect to the internet and broadcast real-time rainfall data. The rainfall sensors were originally intended to tell the vehicle when to turn on its windshield wipers, but the authors decided that these could be used for another purpose of detecting rainfall. The most cost effective way to collect and analyze data are to utilize the resources that are already there, and that is what [1] does. We duplicated this method by using data, database implementations, and programming languages that already exist, then put them together to function for another original purpose. Another study used a high frequency radar in real time to detect what physical atmospheric properties exist during severe rainfall [3]. We thought it would be interesting to see how well they could take an input of our locations and times and output what they thought would be our rainfall data. Most of the studies we encountered were trying to push real time gathering of rainfall data, rather than historical data. For instance, in [4] engineers took cell phones and used them to monitor rainfall in real time. These real time methods used in [1], [3], and [4] are highly experimental and in an early stage of development. Because of this, we decided to make solid, historical rainfall data readily available to use. In this way, in the future, when methods for gathering real time data become more robust, our method can be used to readily scan and extract detailed information from real time data at a glance. Image 10: Rainfall pattern clustering 7 Related Work The ability to quickly retrieve historical rainfall data is extremely important to specific parts of society. Detailed, historical rainfall data can be used to predict future rainfall data, get an idea of what patterns rainfall has, how to better prepare for certain types of weather, and much more. We looked into many past studies of rainfall gathering techniques, and basically came to the conclusion that data is either gathered from the past or in real time. 8 Conclusion Our results were fairly ordinary and not at all surprising. The problem we attempted to solve has been answered before and many of the references show other attempts of solving similar problems. We were able to successfully load two full years of rainfall data into a database and successfully query it for specific results (such as specific locations and times of rainfall). We also were able to apply the K-means algorithm on the data. While our project does not contribute anything new to the field, our results are in agreement with common sense.

5 By performing this project we learned a significant amount about both Data Warehousing and Data Mining. We got to use a collection of useful tools. MySQL was relatively easy to set up but also had a number of difficulties. We needed to make sure the CSV files were properly formatted and a working knowledge of SQL was required. The W3C SQL reference was indispensable toward completing this project. We found that MySQL Workbench was clear and well designed and gaining a working knowledge of the software was indispensable. Orange was extremely easy to use. It is a very power tool for both Data Mining and Data Visualization. In the future if we are ever required to do a similar project Orange will be one of the first products we will consider using. Working in a team is always a valuable experience. It was also interesting to work on an open ended project. While we were behind and only finished last minute. We are still satisfied with the results. [5] Wei Wang, Hui Lu, Tianjie Zhao, Lingmei Jiang, Jiancheng Shi, Evaluation and Comparison of Daily Rainfall From Latest GPM and TRMM Products Over the Mekong River Basin. IEEE Early Access Articles, 2017, Vol. PP, No. 99, PP [6] R. Senthil Kumar, C. Ramesh, A study on prediction of rainfall using datamining technique. IEEE Conference Publications, 2016, Vol. 3 PP. 1-9 [7] W3C SQL Reference [8] Python Documentation [9] Orange [10] MySQL 9 References [1] Carlos T. Calafate, Karin Cicenia, Oscar Alvear, and Juan Carlos Cano, Estimating rainfall intensity by using vehicles as sensors. Conference Publications, 2017, PP [2] National Centers for Environmental Information (NCEI). [3] Haiyin Qing, Yenhsyang Chu, Zhengyu Zhao, Chinglun Su, Chen Zhou, Yuannong Zhang, Observation and analysis of atmospheric rainfall based on the very high frequency radar. IET Journals & Magazines, 2017, Vol. 11, No. 4 PP [4] Hagit Messer, Lior Gazit, From cellular networks to the garden hose: Advances in rainfall monitoring via cellular power measurements. IEEE Conference Publications, 2016, PP

Weather to Climate Investigation: Maximum Temperature

Name: Date: Guiding Questions: Weather to Climate Investigation: Maximum Temperature What are the historical and current weather patterns or events for a location in the United States? What are the long-term