Using Python to Automate Map Exports

Transcription

1 Using Python to Automate Map Exports Don Katnik Maine Department of Inland Fisheries and Wildlife Northeast ARC Users' Conference, 2016 Falmouth, MA DISCLAIMER: I don't work for Python

2 Background Maine wanted to generate a range map for each fish and wildlife species in our 2015 State Wildlife Action Plan. A big job and a good task for automation! In this presentation, I ll explain how I used Python "magic" to automate producing and exporting thousands of ArcGIS maps. A Bunch of Data A Bunch of Maps! NEARC Annual Conference Falmouth, MA Slide 2

3 Presentation Overview Start with what I need, then what I have, and work towards the middle Desired Outputs (maps) Available Inputs (observation data) Intermediate Process-Generated Components Python Scripts Conclusions Inputs Intermediate Components Outputs NEARC Annual Conference Falmouth, MA Slide 3

4 Desired Outputs Species' Maps Identified fish and wildlife species of "conservation need" Across all taxa, habitat types, and areas of Maine Need a range map for each species Inputs Intermediate Components 354 Species Range Maps NEARC Annual Conference Falmouth, MA Slide 4

5 Desired Outputs Terrestrial and Aquatic Maps Based on where the species occurs - Presence/Absence generalized to a Landscape Unit (LU) Two Map Types for each species: Terrestrial (LU = Townships) Aquatic (LU = Sub-watersheds) Inputs Intermediate Components 354 Terrestrial Maps 354 Aquatic Maps NEARC Annual Conference Falmouth, MA Slide 5

6 Desired Outputs Map Format Adobe PDF (familiar to users) Landscape Units "lit up" where species was present Color indicates Data Source User can turn layers (Data Sources) on/off Custom title & legend 354 Terrestrial Maps 354 Aquatic Maps NEARC Annual Conference Falmouth, MA Slide 6

7 Desired Outputs Ability to Revise/Update Additional/updated input (observation) data Additional Species Format/Layout changes Make the Title bigger! I don't like that shade of green! Move the legend! Show the EcoRegional Boundaries! Do those lines have to be gray? It needs to say "DRAFT" in really big letters Inputs Intermediate Components 354 Terrestrial Maps 354 Aquatic Maps NEARC Annual Conference Falmouth, MA Slide 7

8 Available Inputs Observation Data Format Content Data Set(s) GIS Shapefile Polygons MDIFW Essential Wildlife Habitats MDIFW Endangered, Threatened, & Special Concern Data MDIFW Shorebird Areas MDIFW Vernal Pools GIS Shapefile Points MDFIW Fish Data MDIFW Lynx Database Excel Spreadsheet Lat/Long Coordinates ebird Maine Bee Surveys Maine Damselfly/Dragonfly Surveys Access Database Lat/Long Coordinates Maine Mussel Surveys Excel Spreadsheet Township Maine Amphibian/Reptile Atlas Project Excel Spreadsheet Lat/Long or UTM Coordinates Maine Rare Mayfly Surveys Excel Spreadsheet Excel Spreadsheet GIS Shapefile Lat/long or UTM Coordinates or Township or Sub-watershed "Stop" IDs Points "Historical" (Miscellaneous Source) Data Maine Butterfly Surveys Breeding Bird Survey NEARC Annual Conference Falmouth, MA Slide 8

9 Python Scripts Standardizing Observation Data Inconsistent Taxonomy: Strip superfluous words/symbols Identify inconsistent species Hard-code if-then substitutions Location Coordinates Different systems/formats (DD, DDM, DMS; UTM vs lat/long even within the same database) Standardize formats and strip superflous characters with a series of "If Then" Parse UTM vs. Lat/Long Create XY event themes and export to standardized shapefiles NEARC Annual Conference Falmouth, MA Slide 9

10 Python Scripts Standardize Coordinates NEARC Annual Conference Falmouth, MA Slide 10

11 Intermediate Components 15 Input Data Sources (Observations) Standardization 15 Observation Shapefiles Users wanted to know which data source 15 Overlays: Observation Shapefile Landscape Units 15 Output 15 Tables: Shapefiles: Species, LU (presence only) Rows (LUs) = 924 Towns/1,046 Subwatersheds Columns = 354 species 15 Dynamic (Temp) Shapefiles for each Species Values (>325,000) = Presence/Absence Duplicates Geometry, Zero Data Map Template PDF Maps NEARC Annual Conference Falmouth, MA Slide 11

12 Intermediate Components Output Tables One table for each observation data source Each record is a unique Species Landscape Unit occurrence Only presence data Doesn't need to store (repetitively) Landscape Unit geometries NEARC Annual Conference Falmouth, MA Slide 12

13 Python Scripts Populating Output Tables For each Input Data Source: For each Species: Select Observations of the Species from the Input Data Shapefile Overlay Selected Observations with Landscape Units layer For each Landscape Unit where Species was observed: Write Species name and LU code to Output Table Write summary of Species analysis to log file NEARC Annual Conference Falmouth, MA Slide 13

14 Python Scripts Speed Tricks Takes a LONG time to process all data sources and Species Script allows user to specify which data sources to run If overlay output records already exist for a Species, it is skipped (to refresh data for a Species, its previous output records in the table(s) were deleted) For code development, can indicate a "test run" = only one Species will be processed to test if code works Log files = easier debugging NEARC Annual Conference Falmouth, MA Slide 14

15 Python Scripts Speed Tricks Use nested dictionaries to store tabular data NEARC Annual Conference Falmouth, MA Slide 15

16 Python Scripts Speed Tricks Use nested dictionaries to store tabular data NEARC Annual Conference Falmouth, MA Slide 16

17 Intermediate Components ArcGIS Map Template One layer for each Input Data Source Points to a shapefile that is modified for each species Features = Landscape Units (Townships, Subwatersheds) First delete all features Then reload just those features where species was present based on list of LUs for Species based on Data Source LUs = d[ds][species][lutype] NEARC Annual Conference Falmouth, MA Slide 17

18 Python Scripts Speed Tricks Load feature geometries into a Python dictionary NEARC Annual Conference Falmouth, MA Slide 18

19 Intermediate Components ArcGIS Map Template TITLE = SPECIES NAME This "text element" in the template layout has a unique name that allows Python to find and manipulate it (i.e., change the text string) CUSTOM LEGEND Each of these composite graphic/text elements is the legend item for an observation data source. By default they are "docked" off the page. For each Species, Python finds the legend items for its observation data sources and moves them onto the page to build a custom legend. NEARC Annual Conference Falmouth, MA Slide 19

20 Python Scripts Making the PDF Maps For each Species: Delete LU features (Townships or Subwatersheds) from shapefiles in MXD For each Input Data Source: Get (now empty) shapefile Query 3-level Python Dictionary for records: Load list of LUs into the empty shapefile = d[data Source][Species][LuType] Customize map layout Title (Species name) Build custom legend Toggle layers on/off depending on which have data for the Species Export ArcMap Layout to PDF NEARC Annual Conference Falmouth, MA Slide 20

21 Python Scripts Speed Tricks Takes a LONG time to process all Species If PDF map files already exist for a Species, it is skipped (so to refresh maps for a Species, the existing PDF files must be deleted) Load Landscape Unit feature geometries into a Python dictionary Load GIS overlay output tables into a nested Python dictionary Log files = easier debugging NEARC Annual Conference Falmouth, MA Slide 21

22 Conclusions I have refreshed the entire set of maps several times Have frequently replaced maps for select Species Would have been almost impossible without script automation Generating log files as the scripts processed data was invaluable for debugging problems And the peasants rejoiced and gave praise to Python! NEARC Annual Conference Falmouth, MA Slide 22

23 The Obligatory "Got Questions?" Dog Pic I nodded off. What did you say after, "Today I'm here to talk about "? NEARC Annual Conference Falmouth, MA Slide 23

24 Conclusions, Cont. Recall: GIS overlay results were stored in tables instead of a big, fat shapefile with lots of negative values Minimal file sizes Produced desired output with reasonable processing time Easy to add additional data sources and Species Map template easy to change Users sometimes wanted the saved shapefiles for a Species (they don t exist) Changing a map required re-running the script (cannot just edit an MXD) Multi-species queries require custom scripts Not intuitive to most users; requires detailed documentation to ensure someone else could use the process NEARC Annual Conference Falmouth, MA Slide 24