The output from my MapReduce job is a text file with each line containing a set of coordinates and a floating point value in the following format:

Transcription

1 Cloud Computing Project Purpose For this project, my goal was to create heat maps (a 2D map where values are represented by gradients) for analysis of where the most government spending occurred in the state of Illinois. I used tab-delimited spending data for contracts from the 2010 fiscal year with the Departments of Defense, Agriculture, and Justice ( Hypothesis At the onset of this project, I assumed that the most spending for defense contracts would be around the Chicago area, since there are many defense contractors in both the Chicago metropolitan area and the surrounding suburban areas. For the Department of Agriculture, I figured the data would be spread out across the state, focused on agricultural areas rather than areas like Chicago. The Department of Justice would most likely have large contracts in areas where federal prisons are located, such as the Federal Correctional Institute in Pekin. Procedure I used Hadoop in the Karmasphere environment through the NetBeans IDE to map and reduce the raw spending data rows. I ran separate jobs for each of the three agencies on which I conducted the analysis. The DollarsObligatedZipCodeMapper reads from column 42 (RecipientZipCode) and column 10 (DollarsObligated), and maps the amount of dollars obligated by the government agency in the contract to the zip code of the recipient. I used Java s regex to verify the zipcodes, since I noticed a few rows either omitted them or had words like Illinois written instead. I used zipcodes because they were the simplest and most specific piece of data available from which to obtain geographical coordinates of each contractor. The FloatSumReducer does more than its name describes. Not only does it calculate the sum of all contracts in a given zipcode, but it also uses Google s Geocoder API to look up the zipcode and obtain its associated geometrical coordinates (e.g. This reducer does take a while to process everything, and Google has a usage limit of somewhere from 15,000 to 50,000 daily queries per IP address, so I was careful to run the actual job a minimal number of times. This may have resulted in a minor loss of information, as the Geocoder application could have timed out at certain points due to attempted concurrent access or latency. However, I feel that the mapper retained the necessary information that would allow me to generate a proper heatmap. The output from my MapReduce job is a text file with each line containing a set of coordinates and a floating point value in the following format: Latitude,Longitude DollarAmount ,

2 I ended up deferring the actual heatmap generation to an external Java application I wrote myself (HeatMapAnalysis), since I couldn t think of a reasonable way to incorporate it into the MapReduce job. Due to time constraints, I simply had the program generate a heatmap as a PNG with the same dimensions of a map of Illinois I had prepared. The positions of the heatmaps were skewed as well, but I managed to correct them in Photoshop and apply a blur effect to give them a better appearance. I would then copy and paste the heatmap onto the aforementioned map of Illinois and save it for each of the three agencies that I analyzed. The heatmap ranges from Green (lowest money spent) to Yellow (median money spent) to Red (highest money spent). Results My results were not entirely what I expected. Since I focused on government contracts rather than all aspects of government spending, I received a narrow range of expenditures. For the Dept of Defense, I was right in that a large amount of spending went towards defense contractors in Chicago, but Rockford also had a great deal of spending in its area. Additionally, some other smaller areas like the Quad Cities had somewhat small contracts. The Dept of Agriculture was mostly focused in Chicago and the Champaign area, which in retrospect makes sense due to Chicago being home to major food suppliers, and Champaign having Kraft foods as well as being a nexus for agricultural studies. The Dept of Justice had some respectable spending for the prisons near Peoria and Springfield, however, Chicago and what I assume to be the Waukegan area have greater endowments, possibly due to the Cook and Lake county court systems and county jails. Conclusion I am satisfied with the results that I have obtained from this project. I was half-right about my guess for the Departments of Defense and Justice, and I was surprised by the actual result for the Dept of Agriculture spending. Based on my work with this project, I felt that Hadoop was definitely useful for the task of geocoding as part of a MapReduce job. Due to the fact that the reducer has to reference an online URL, I am not sure if this job can be run on a cluster isolated from the internet. In my search for implementations of geocoding in Java, I did find JGeocoder ( but it requires installation of a considerably large offline database containing geometric coordinates, which I found undesirable for the circumstances of this project. This is my first time using heat maps as a method of analysis, and I figured they would be a great method of analyzing the USA Spending data. The outputs from my analysis will be on the next pages.

3 Dept of Defense Spending (2010)

4 Dept of Agriculture Spending (2010)

5 Dept of Justice Spending (2010)