GARP 0544 Cookbook 25 March 2013 GIS Data MassGIS

Transcription

1 GARP 0544 Cookbook 25 March 2013 GIS Data MassGIS Overview 1. What is MassGIS? 2. File Geodatabases, Feature Classes and Shapefiles 3. Downloading MassGIS Data: General Principles 4. Downloading MassGIS Data: Vector Data 5. What Are You Actually Downloading? 6. Review: Downloading MassGIS Vector Data 7. Oliver vs. File Listing 8. Homework Assignment #5 This week we will explore online sources of GIS data. There are literally hundreds of websites and unlimited amounts of data available for free downloading, mostly from state and federal agencies all around the world. This week we focus on MassGIS the GIS data repository for the Commonwealth of Massachusetts. But, the basic sequence for downloading GIS data is always the same: 1. You find what you want (that s the hard part) and download it. 2. The data usually comes in some zipped or compressed format. 3. You open the data in ArcMap and hopefully you got what you were looking for. Otherwise, try again! MassGIS is simple and efficient perhaps not as fancy as some of the other websites. The download speeds are usually very good and the data quality is excellent especially the data documentation. So, go ahead and start downloading data! But first, read and work through this cookbook and play particular attention to the data organization described in Section 3. You will be downloading a lot of data layers, so please start organized otherwise it will get really messy in a few weeks! GARP 0544 (Spring 2013) Page 1 Dr. Carsten Braun

2 1) What is MassGIS? MassGIS: MassGIS is a state agency funded by the tax payer. It s free, but part of the mandate of the Commonwealth of Massachusetts. We, as tax payers, are paying for this service. You will make extensive use of the free data provided by MassGIS in your projects for the remainder of the semester. Other states provide similar services, but the MassGIS website is exceptionally good, comprehensive, and frequently updated. Cities and towns often provide online access to their GIS data, but usually only for viewing, not for downloading. This has become an advertisement tool for communities to attract new businesses. This week we will explore the MassGIS website and familiarize ourselves with the available data and services. Then, we will download different data sets and make maps. 2) File Geodatabases, Feature Classes and Shapefiles Storing geospatial data (= spatial and attribute data) is a trickier than storing just attribute data (which is all that MS Excel has to do). Spatial Data Where is it? (= Location, latitude and longitude) What is it? (= Geometry, point, line, or polygon) What s next to it? (= Topology, adjacency, etc.) Examples of spatial data: the location and geometry of a wetland (polygon), a river (polyline), or utility pole (point). Spatial data tells us where something is located, but not what it is. Attribute Data What are its characteristics? Attribute data tells us what a feature is (i.e. its characteristics), but not where it is located. Attribute data are stored in a database table, hence the term attribute table is used in GIS Examples of attribute data: the height of a mountain and its rock type, the bio-diversity of a wetland, and whether a road is paved, dirt, private, etc. The power of a GIS is that it can handle, display, and analyze both types of data together. You can ask Where is something? and What does it do? You can imagine two different approaches for storing geospatial data. GARP 0544 (Spring 2013) Page 2 Dr. Carsten Braun

3 Plan A: The Keep Everything Separate Approach Here, everything (spatial data, attribute data, pictures, raster data, etc.) is kept separate and stored as seperate files, grouped as families, and linked by so-called index files. That makes some sense, because it neatly separates, for example, the spatial data from the attribute data. That s what GIS users are (still) do right now using shapefiles and we have a special software program called ArcCatalog to help us organize the many individual files. Shapefiles have an obvious disadvantage they are difficult to keep organized. On the other hand, each file itself is simple and you can always figure out what is going on. Plan B: The Kitchen Sink Approach Here, everything (spatial data, attribute data, pictures, raster data, etc.) is stored together in a single, very complicated database table. That sounds appealing, since your GIS is located inside one single file and all you need is a GIS software to read that file. In fact, that s where we are with ArcGIS10.1. Esri has introduced a new data format called a geodatabase to serve effectively as the kitchen sink. GIS users are gradually changing over to this new data format. Geodatabases have some disadvantages: they tend to use up a lot of space on your computer and are so complex that it is almost impossible to figure out what is going on inside the geodatabase. Geodatabases are clearly the future! However, shapefiles are still very common in the GIS world and will not go away in the foreseeable future thus we are dealing with a bit of a messy situation in terms of GIS data formats. 2.1) File Geodatabases and Feature Classes The C:\ESRIPress\GIST1\Data folder contains a series of folders and file geodatabases, for example the UnitedStates.gdb and the World.gdb. Uou can think of a file geodatabase as a container holding all kinds of GIS data sets. For example, vector data are stored inside geodatabases as so-called feature classes. However, the structure of geodatabases is so complicated and complex that you need ArcCatalog to look inside a geodatabase. Try it in Windows Explorer for fun all you see incomprehensible chaos! GARP 0544 (Spring 2013) Page 3 Dr. Carsten Braun

4 The figure below shows the feature classes contained in the file geodatabase UnitedStates.gdb. Each feature class is one vector data set (point, line, or polygon). ArcCatalog allows you to preview the geography and attribute table of each feature class so you can get an idea of what the feature class contains before adding it to ArcMap. Using Add Data you can add each feature class to your map as a new map layer. Features Classes contained in the file geodatabase United States.gdb There are three types of geodatabases: File geodatabases Stored as folders in a file system. Each dataset is held as a file that can scale up to 1 TB in size. The file geodatabase is recommended over personal geodatabases. Personal geodatabases All datasets are stored within a Microsoft Access data file, which is limited in size to 2 GB. ArcSDE geodatabases Stored in a relational database using Oracle, Microsoft SQL Server, IBM DB2, IBM Informix, or PostgreSQL. These multiuser geodatabases require the use of ArcSDE and can be unlimited in size and numbers of users. Always use a file geodatabase! GARP 0544 (Spring 2013) Page 4 Dr. Carsten Braun

5 Additional Information ) Shapefiles Most of the GIS data from MassGIS is distributed as so-called shapefiles thus we need to learn how to handle shapefiles. A shapefile has to include (at minimum): The geographic data of the spatial feature. The attribute data of the spatial feature. An index, linking the geographic data to the attribute data. Therefore you can: 1) Look at a shapefile as a map layer (= map the geographic data). 2) Look at a shapefile as a table (= view the attribute data). 3) Use the attribute data to change how the associated geographic data are displayed on the map. Example The file format for MS Word is *.doc = a MS Word document is stored on your computer as a *.doc file. The file format for GIS data is a shapefile, meaning that GIS data are saved on your computer as shapefiles. As mentioned above, shapefiles are being gradually replaced with a new file format called geodatabases. Here is a screen-shot of a folder called C:\GIStutorial\UnitedStates. There are 28 files visible in Windows Explorer but only actual 4 shapefiles: States, Counties, Cities_dtl, and Cities. GARP 0544 (Spring 2013) Page 5 Dr. Carsten Braun

6 Here is a close-up and you see 7 individual files, but only 1 actual shapefile: States. That s a bit weird but that s the way it is. A shapefile is actually not a single file, but a family of 3 to 7 individual files. There are three mandatory files for each shapefile: States.shp The shapefile proper The geographic data: shape + location = geometry of the features States.dbf A data base table with associated attribute data of the features States.shx A spatial index file to link the geometry of features to their attributes There are several optional files for each shapefile:.sbn/.sbx Additional index files as needed.prj The projection file.shp.xml meta data for the shapefile (= information about the data) We are used to a different situation! For example, you type a paper in MS Word and save it as a single *.doc file. This one single file contains all your text, all your figures, all your tables everything. That single file you can attach to an and, for example, send your professor. Well, GIS data are handled differently. The States shapefile is not a single file, but a family of seven files even though we refer to it simply as the States shapefile. Therefore you need to combine or zip the individual files together as a single archive for sharing or downloading. Example with ArcCatalog Below is a screen-shot of the C:\GIStutorial\United States folder, but this time viewed in ArcCatalog. ArcCatalog manages and organizes the GIS data. ArcCatalog only shows you the States.shp file and hides the rest of the files that make up the shapefile family. This makes it easier to organize your files. You only deal with 1 file and ArcCatalog takes care of the rest. GARP 0544 (Spring 2013) Page 6 Dr. Carsten Braun

7 GARP 0544 (Spring 2013) Page 7 Dr. Carsten Braun

8 2.3) Who Cares? Good point we have to! GIS means using and mapping geospatial data and therefore we need to deal with these different file types. In a few years shapefiles will be obsolete, but in the meantime we need to be able to use geodatabases/feature classes and shapefiles. Most MassGIS data is distributed as shapefiles. You can easily import shapefiles as features classes into a geodatabase in ArcCatralog. You can easily export a feature class as a shapefile using ArcCatalog. Fun with your new language! GIS data = geospatial data, stored as feature classes inside file geodatabases or as shapefiles. Feature classes (or shapefiles) are organized in ArcCatalog. Feature classes (or shapefiles) are added as map layers to a map in ArcMap. The map you create in ArcMap does NOT contain the feature classes (or shapefiles) underlying each map layer, but a link back to the location where the actual feature classes (or shapefiles). GIS map and GIS data are independent, but linked! 3) Downloading MassGIS Data: General Principles The following general principles of data downloading apply to any online source of data the key is to be organized and to know what you are downloading before you download it! Data Organization Data organization is crucial! You will be downloading a large amount of data layers from MassGIS in the coming weeks so please follow the data organization suggested below to make your life easier! Your USB flash drive is already organized correctly: you created all your necessary folders, including a folder called \garp0544\data. All data you download from anywhere is saved into this folder! Do NOT = NEVER save any data into any other folder! Never save anything to these local computers in the lab it will be erased! I cannot overemphasize how important this is please do yourself a favor and follow these instructions you will understand the reasoning very soon! It is literally impossible to organize a mess of data once it is messy and I can guarantee that you will run into severe problems! You have been warned! GARP 0544 (Spring 2013) Page 8 Dr. Carsten Braun

9 Screen Capture: USB flash drive and folder organization suggested for this course. Above is the folder organization we are using: There is a folder for each week (Week 1 to Week 16) There are three folders for the three projects There is a folder for the 13 cities project There is a folder for course materials There is a folder called scratch for files we don t really need to keep, but need to save at least temporarily. Most important: there is a dedicated folder for your GIS data GARP 0544 (Spring 2013) Page 9 Dr. Carsten Braun

10 Screen Capture: Organization of your \data folder. 3.1) Create Your Filegeodatabase We already created a file geodatabase to store feature classes that we create or download over the next few weeks. Otherwise, here are the instructions: 1. Start ArcMap and open the ArcCatalog window. 2. Create a folder connection to your USB flash drive, to the \garp0544_spring2013\data folder. 3. In ArcCatalog, browse to the \garp0544_spring2013\data folder. 4. Click on the folder to select and highlight it. 5. Right-click, select New, File Geodatabase, and name it: garp0544cbdata (replace cb with your initials). GARP 0544 (Spring 2013) Page 10 Dr. Carsten Braun

11 3.2) Create The Folders for Your Shapefiles Now you need to create a series of dedicated folder to store the shapefiles that we create or download from MassGIS over the next few weeks. Create three sub-folders inside the \garp0544_spring2013\data folder: \ma_firestations \ma_topomaps25k \ma_towns Always! USE exactly these names! Do NOT use blanks or capitals! Always! Anytime you create or download a shapefile: first create a dedicated folder for the shapefile with a reasonable name. 3.3) More MassGIS Data Downloading Rules Data Credits See for the correct text you have to include when using MassGIS data: Data are from Office of Geographic Information (MassGIS), Commonwealth of Massachusetts Information Technology Division Always! You have to include this sentence on your map when you use MassGIS data otherwise you are committing plagiarism. Available Data Layers (for download) See The data layers are organized into Image/Raster data (at the top) and Vector Data (below, scroll down for the complete listing). Or, select from the Data Category links listed at the top of the page. Each digital dataset name links to a complete data layer description and a link to that data layer's free download page. Always! Please read and understand the data layer description before downloading anything! GARP 0544 (Spring 2013) Page 11 Dr. Carsten Braun

12 3.4) Data Download Check List Before you download data (from MassGIS or otherwise), ask yourself the following questions: 1) Do I really need this data layer? 2) Did I create the necessary folder(s)? 3) Do I understand what I am downloading? 4) Do I know how to correctly cite/source this data layer? 5) Did I read and understand the data layer description? Always! Proceed with download ONLY when the answer to all 5 questions is YES! 4) Downloading MassGIS Data: Vector Data Mapping Task Create a map showing the fire stations in Massachusetts and analyze if there are any parts of the state more than 3 miles from a fire station. Unfortunately, we cannot finish this map just yet because we have not covered the Buffer Tool in ArcGIS. But we can make the map and use it in a few weeks when we cover the Buffer Tool. Our map has two layers: Base Layer: Analysis Layer: Town Lines Fire Stations You can create similar maps for police stations, hospitals, ice rinks, prisons, libraries, schools, etc. MassGIS provides vector data layers for these spatial features all we need to do is to download the two data layers we need for our map. Of course, you could also use a base map from ArcGIS Online as your base layer. 4.1) Downloading Vector Data from MassGIS 1) Find the data layer on the MassGIS website. 2) Go through your Data Download Checklist (Section 3.4) and review the information about the data layer. 3) Click on Download this layer (usually found in the upper-left corner). 4) Save the file into your dedicated folder. GARP 0544 (Spring 2013) Page 12 Dr. Carsten Braun

13 All MassGIS Vector Data are distributed in self-extracting zip archives. Zipping compresses the data, making data storage easier and data downloads faster. Each shapefile can be up to 7 individual files, plus optional legends. It is therefore easier to package all files comprising the shapefile and its accessories into one single file (= zip archive) for download. On the computer, you unzip the zip archive into its individual files. This is easy, but the specific steps differ from computer to computer, operating system to operating system, and web browser to web browser. Download and Unzip the Fire Stations 1) Goto 2) Read the data layer information. 3) Click on Download this layer. 4) Save the file (= the zip archive) into your \garp0544_spring2013\data\ma_firestations folder. Depending on your computer, operating system, or browser: files may be downloaded automatically into a default directory. In this case: find the file and cut/paste into your proper folder \garp0544_spring2013\data\ma_firestations. 5) Double-click on the file, select Run if prompted. Or, right-click and select Extract All. Again, the precise sequence here depends on your computer, operating system, browser, and third-party programs such as 7-Zip etc. 6) This Step is Crucial! Browse to the folder containing the zip archive you are unzipping. The correct folder has to be displayed under Unzip to folder. 7) Select Unzip and 12 files are unzipped and uncompressed from the zip archive into your folder. 8) Browse to the garp0544_spring2013\data\ma_firestations folder and confirm! GARP 0544 (Spring 2013) Page 13 Dr. Carsten Braun

14 Screenshots for Windows 7 GARP 0544 (Spring 2013) Page 14 Dr. Carsten Braun

15 Screenshots for Windows XP GARP 0544 (Spring 2013) Page 15 Dr. Carsten Braun

16 Download and Unzip the Town Lines 9) Goto and download the data layer Community Boundaries (Cities and Towns). 10) Save the zip archive into your \garp0544_spring2013\data\ma_towns folder. 11) Extract / unzip the file. 12) Browse to the garp0544_spring2013\data\ma_towns folder and confirm! Unfortunately, these steps may differ from computer to computer! On some computers you can right-click on the zip-archive and select Extract to Here. You need to test the procedure on your computer! Regardless of the exact procedure you have to ensure that you extract or unzip the archive into the proper folder location on your USB flash drive! 4.2) Making the Map Now that we have the data making the map is easy! 1. Start ArcMap and make the necessary folder connections. 2. Add the two data layers to your map. 3. Open the Search window and search for: Buffer tool. 4. Open the Buffer tool, make you click-on Show Help. That opens a context-sensitive Help window that walks you through the different inputs and options. Click on Tool Help to get additional answers and explanations. 5. Execute the Buffer Tool. 6. Switch to Layout View and add all the necessary map elements, adjust colors, labels, layer names, etc. 7. Confirm relative pathnames and save your map into your \week10 folder. GARP 0544 (Spring 2013) Page 16 Dr. Carsten Braun

17 5) What Are You Actually Downloading? It is worth looking at what we just downloaded from MassGIS. We downloaded a zip archive called towns.exe from MassGIS. Then we extracted / unzipped the file and ended-up with 36 additional files. 3 Shapefiles TOWNS_POLY Towns as 631 polygons TOWNS_POLYM Towns as 351 multi-part polygons TOWNS_ARC Town outlines as 1428 polylines or arcs 5 Layer Files Layer files are covered in the GIS Tutorial. Layer files, in essence, make the underlying shapefiles (or feature classes) look good by saving their symbology (the colors, sizes, outlines, etc.). I m not a big fan of layer files I prefer to create my own symbology for my maps layers as needed for each map I create. 2 Tables One table contains population data; the other table contains area codes. Remember! The GIS data you downloaded are the 3 shapefiles. The Layer files are nice, but only provide you with the symbology for the shapefiles! MassGIS often includes Layer and Legend files for your convenience, but it is generally better to create your own symbology that is relevant to your map. GARP 0544 (Spring 2013) Page 17 Dr. Carsten Braun

18 5.1) Example: MassDOT Roads 1. Goto and download the MassDOT road layer for Westfield. Be sure to create a dedicated folder for your data layer and unzip the downloaded file. 2. Right-click in ArcCatalog, select Refresh and your new folder and new shapefile called eotroads_329 should appear. 3. Add the file to your map and zoom into Westfield. Pretty boring as all roads are shown as lines using the same color and width. 4. Go back to the MassGIS website and read over the data layer description. Note under Attributes is a description of the attribute table field called CLASS that designates each road based on its class and access: 1 - Limited Access Highway 2 - Multi-lane Highway, not limited access 3 - Other numbered route 4 - Major road - arterials and collectors 5 - Minor street or road (with Road Inventory information, not class 1-4) 6 - Minor street or road (with minimal Road Inventory information and no street name) Great now we can make a choropleth map and use this field to determine the color and size of our lines. 5. Go back to ArcMap and create a choropleth map showing the color of each road as a function of its class. Select the Symbology tab, Quantities, Graduated colors, Values = Class, use 6 classes, and hit OK. 6. Now modify each line color and line width so that the symbol is more appropriate for each road class. 7. Once done, right-click on the eotroads_329 map layer, select Save as Layer File, and save your layer file into the folder where you eotroads_329 shapefile is located. You can use this Layer file in any future map that includes the road layer for Westfield! GARP 0544 (Spring 2013) Page 18 Dr. Carsten Braun

19 6) Review: Downloading MassGIS Vector Data Most vector data from MassGIS are distributed as so-called shapefiles thus we need to learn how to handle shapefiles! Raster data are a different story more about that next week. A shapefile is actually not a single file, but a family of 3 to 7 individual files. The States shapefile is not a single file, but a family of seven files even though we refer to it simply as the States shapefile. Therefore you need to combine or zip the individual files together as a single archive for sharing or downloading. Data Organization Data organization is crucial! You will be downloading a large amount of data layers from MassGIS in the coming weeks so please follow the data organization suggested below to make your life easier! Your USB flash drive is already organized correctly: you created all your necessary folders, including a folder called \data. Anytime you create or download a shapefile: first create a dedicated folder for the shapefile with a reasonable name. Do NOT use blanks or capitals! Data Credits Data are from the Office of Geographic Information (MassGIS), Commonwealth of Massachusetts Information Technology Division Zip Archives MassGIS Vector data are distributed in self-extracting zip archives. On the computer, you unzip the zip archive into its individual files. This is easy, but the specific steps differ from computer to computer, operating system to operating system, and web browser to web browser. On some computers you can right-click on the ziparchive and select Extract to Here. You need to test the procedure on your computer! Regardless of the exact procedure you have to ensure that you extract or unzip the archive into the proper folder location on your USB flash drive! GARP 0544 (Spring 2013) Page 19 Dr. Carsten Braun

20 7) Oliver vs. File Listing There are two different ways to access, view, and download MassGIS data. Oliver Oliver is the MassGIS Online Data Viewer and works great for viewing and extracting data. The Data export wizard includes a great Help document that walks you through the exporting procedure in 4-steps. Data Layer Listing This is the more traditional, old-school way of getting your data. I prefer it for a few reasons: It tends to be faster and does not rely on a web browser-based application. It emphasizes the underlying data structure. It is robust and works on all platforms. GARP 0544 (Spring 2013) Page 20 Dr. Carsten Braun

21 8) Homework Assignment #5 This assignment reinforces and expands the skills you have learned this week. MassGIS is the GIS data repository for the Commonwealth of Massachusetts in essence a website where you can download GIS data for Massachusetts. URL: The vector data are typically downloadable as zipped shapefiles, whereas the raster data (e.g. aerial photographs, topographic maps, etc.) are downloadable in the MrSID file format. Part 1: MassGIS Exploration Here are 7 questions to guide your exploration of the MassGIS website. You will also explore similar websites from Connecticut, New Hampshire, Vermont, Harvard University, and Tufts University. Summarize your explorations in a ~2 page report that includes the answers to the 7 questions. 1. Explore the data available at MassGIS. List 5 or more data layers that you would find especially interesting. (= Create your own Top-5 List). 2. When using MassGIS data what source credit has to be included on the map? (Type or copy/paste the specific text required) 3. Explore the GIS data available for Connecticut at How does the CT website compare to the MassGIS website? 4. Explore the GIS data available for New Hampshire at How does the NH website compare to the MassGIS website? 5. Explore the GIS data available for Vermont at How does the VT website compare to the MassGIS website? 6. Explore the Tufts Geospatial Data Repository at 7. Explore Harvard s offerings at GARP 0544 (Spring 2013) Page 21 Dr. Carsten Braun

22 Part 2: GIS Mapping Now it is time to download data layers and to make three meaningful and professional maps. Map 1 Map 2 Map 3 A map of the whole state, using vector data layers. Download at least three vector data layers (1 point, 1 line, 1 polygon). Map these data layers as a meaningful map. Similar to Map 1, but showing a close-up of a city or town in Massachusetts. For example, show the fire station(s), school(s), town hall, library, and roads. A map of all Colleges and Universities in Massachusetts/ Use a different symbol depending on whether the institution is public or private. Use a base layer from ArcGIS Online as your base layer. Hints/Comments Include all required map elements and appropriate data credits on both maps. You can download whatever vector data sets you desire for Map 1 and Map 2 as long as you download at least 1 point feature, 1 line feature, and 1 polygon feature. Make sure your legends are meaningful and appropriate. Deliverables Please submit professional, well-written report using proper English language and professional formatting and layout. Think in terms of using this report as a sample of your work for a job interview. Include: Your report of your MassGIS Exploration (~2 pages). A step-by-step documentation of your mapping procedure Think in terms of writing a cook book recipe that a similarly-trained GIS user can follow along to repeat your mapping. I need to be able to understand your methods to assess your work. Use a numbered list to organize your recipe. Be sure to include information and documentation as to which data layers you have downloaded from MassGIS (~3 pages). Your 3 printed maps. As always, include a cover page and page numbers in the page footer. Your report will have about 9 pages (cover page, ~2 pages for the MassGIS exploration, ~3 pages for the documentation, and 3 pages for the maps). Please note: the maps, the exploration questions, the documentation, professionalism and attention to detail are all equally important in terms of your assignment grade. Contact me for help or clarification of this assignment or my expectations as needed. Bonus Challenge! Looking for a little challenge? Make a 4 th map by downloading the USGS topographic map(s) that cover our campus! GARP 0544 (Spring 2013) Page 22 Dr. Carsten Braun