Data Basin ArcGIS Geoprocessing Tools Documentation

Transcription

1 Data Basin ArcGIS Geoprocessing Tools Documentation Release 0.2 Conservation Biology Institute July 22, 2014

2

3 Contents 1 Tabulate Tool Available Summary Methods Inputs Outputs Error Handling Messaging messaging.py Utilities PathUtils.py ProjectionUtilities.py FeatureSetConverter.py Requirements 17 5 Installation - ArcGIS 10.2.x Pre-packaged versions Packaging and deploying yourself Testing 21 7 Known Limitations 23 8 Authors 25 9 License 27 Python Module Index 29 i

4 ii

5 These geoprocessing tools were created to help support spatial analysis and other operations in Data Basin. Source code available in Bitbucket. These tools are intended to be deployed as geoprocessing services on an ArcGIS Server that also hosts map services (their data must be hosted locally). Within Data Basin, these services are accessed via server middleware to route to and control the number of jobs per ArcGIS server. Custom-built client code (HTML/JS/CSS) is used to provide the user interface to these tools, and manages generation of input parameters, and display of output results. However, most of the internal functions only rely on arcpy and access to the input data, and thus could be executed from other environments. Refer to the specific tools below for more information: Contents 1

6 2 Contents

7 CHAPTER 1 Tabulate Tool This tool tabulates various summary values for feature or raster datasets within an area of interest. The area of interest can be represented as one or more points, lines, or polygons (limited to one type of geometry per analysis). This tool creates a custom Albers Equal Area (WGS84 datum) projection centered over the area of interest to use as the standard throughout processing; however, the native projection of the target raster dataset will be used if it is a valid projection for calculating areas, such as Albers Equal-Area, Lambert Azimuthal Equal-Area, or UTM. For raster analysis, the tool uses one of two methods: 1) approximate: the area of interest is converted to a raster dataset with the same resolution as the target raster (pixel calculations are not based on partial pixels); thus it is necessary to compare the area of interest in pixels against the summary area returned for the target raster. This method is used when the area of interest is represented by points or the number of pixels in the extent of the area of interest is higher than optimal for precise method (>50,000 pixels). 2) precise: the raster is extracted to the extent of the area of interest in its native projection, and then a fishnet feature class is created that matches it. This fishnet is then intersected with the area of interest, and proportional areas of overlap area calculated as weights for each pixel. These weights can then used for either area weighted statistics. 1.1 Available Summary Methods Feature layers area or length and count of features inside area of interest area or length of features that intersected area of interest (the total area or length of the original feature both inside and outside area of interest) area or length and count of features by unique attribute values area or length and count of features by classes of a continuous attribute statistics of a continuous attribute inside area of interest: MIN, MAX, SUM, MEAN Raster layers Note: MEAN is always weighted by the polygon area, line length, or point count of the target features within the area of interest. area and pixel count of area of interest in resolution of target raster area and pixel count of raster inside area of interest area and pixel count of unique values of a raster or raster attribute inside area of interest area and pixel count of classes of a continuous raster or raster attribute inside area of interest 3

8 statistics of raster or continuous attribute inside area of interest. Valid statistics are: MIN, MAX, SUM, MEAN, STD (standard deviation). Note: If the precise method is used, MEAN is weighted by the proportion of each pixel occupied by the area of interest if it is a polygon, or by the proportional length if area of interset is a line. 1.2 Inputs featuresetjson: Area of interest represented as an ArcGIS FeatureSet in JSON format: } "fields": ["alias": "OBJECTID", "type": "esrifieldtypeoid", "name": "OBJECTID"}], "geometrytype": "esrigeometrypolygon", "features": [ "geometry": "rings": [ [ [ , ], [ , ], [ , ], [ , ], [ , ], [ , ], [ , ], [ , ] ] ] "attributes": "OBJECTID": 3} } ], "spatialreference": "wkid": ,"latestWkid": 3857} configjson: The list of map services, layers, and summary methods: "services":[ "serviceid":"test","layers":[ "layerid":0 "layerid":0,"attributes":["attribute":"name"}] "layerid":0,"attributes":["attribute":"pop2000", "statistics":["min","max"]}] "layerid":2,"attributes":["attribute":"pop2000","classes":[[0,1000],[1000,10000],[1000 "layerid":3 "layerid":5 "layerid":5,"classes":[[0,300],[300,310],[310,400]] "layerid":5,"statistics":["min","max","mean","sum"]} ]} ]} For each map service, provide the serviceid (from the map service URL, this is /arcgis/rest/services/<serviceid>/mapserver), and the layer configuration. For each layer, provide the layerid (this can be determined from looking at the list of layers for the map service in ArcGIS REST API). If no other parameters are given for layer, only the total area or length and count of features inside area of interest, and total area or length and count of features intersecting the area of interest will be returned. 4 Chapter 1. Tabulate Tool

9 Feature layers: To summarize by unique values of an attribute, simply include that attribute in the list of attributes: "layerid":0, "attributes":["attribute":"name"}]} To summarize by classes of an attribute, include the attribute and list of class value ranges (greater than or equal to first value, and less than second value): "layerid":0, "attributes":["attribute":"name", "classes":[ [0,10], [10,20], [20,30] ]}]} To return summary statistics of an attribute, list the desired statistics: "layerid":0,"attributes":["attribute":"pop2000","statistics":["min","max","mean","sum"] }] Note: statistics option is mutually exclusive of above options Raster layers: Categorical rasters will be summarized by unique value if no additional parameters are provided, continuous ones will not: "layerid":3} To summarize by classes of the raster, simply include class ranges at layer level: "layerid":5, "classes":[ [0,300],[300,310],[310,400] ]} To return summary statistics of raster, simply include statistics at layer level: "layerid":5, "statistics":["min","max","mean","sum","std"]} Attribute-level summaries are same as above 1.3 Outputs During execution, the tool will add a progress message for each completed layer and service. PROGRESS [PERCENT_COMPLETE] resultsjson: JSON results follow similar format as configjson above. Key concepts: The format is: Very little is returned if no intersection is found. Generally only count properties will be returned in this case. Areas and lengths are returned using the general intersectionquantity properties. Use the geometrytype properties to determine what units these represent. Quantities will not be returned for points. An important distinction is made between intersected and intersection results for features: Intersection: the portion of the features WITHIN the area of interest. This will be in the units of the intersection. Intersected: the original features that intersected the area of interest, INCLUDING the area of length inside and outside the area of interest. This will be in the units of the original intersected features. This is useful for calculating the percentage of the original features that are within the area of interest. Results for examples above: 1.3. Outputs 5

10 "area_units": "hectares", #area values are always in hectares "linear_units": "kilometers", #linear values are always in kilometers "sourcegeometrytype": "polygon", #point, line, or polygon "services": ["serviceid": "test", "layers": [ #a point feature layer "layerid": 0, "intersectiongeometrytype": "point", #will be point, line, polygon, or pixel "intersectedcount": 2, #number of features that INTERSECTED area of interes "intersectedgeometrytype": "point", "intersectioncount": 2 #number of featues WITHIN area of interest "layerid": 0, "intersectedgeometrytype": "point", "intersectedcount": 2, "attributes": [ #a categorical attribute "attribute": "NAME", "values": [ "intersectedcount": 1,"intersectionCount": 1,"value": "Avondale "intersectedcount": 1,"intersectionCount": 1,"value": "Goodyear ] } ], "intersectiongeometrytype": "point", "intersectioncount": 2 "layerid": 0, "intersectedgeometrytype": "point", "intersectedcount": 2, "attributes": [ #a continuous attribute "attribute": "POP2000", "statistics": "MAX": 35883, "MIN": } } ], "intersectiongeometrytype": "point", "intersectioncount": 2 #a polygon feature layer "layerid": 2, "intersectiongeometrytype": "polygon", "intersectedgeometrytype": "polygon", #quantities are hectares for polygon geometry type, kilometers for line, and "intersectionquantity": , "intersectedquantity": , "intersectedcount": 1, "attributes": [ 6 Chapter 1. Tabulate Tool

11 #a continuous attribute "attribute": "POP2000", "classes": [ "class": [0,1000], "intersectedquantity": 0, "intersectedcount": 0, "intersectionquantity": 0, "intersectioncount": 0 "class": [1000,10000], "intersectedquantity": 0, "intersectedcount": 0, "intersectionquantity": 0, "intersectioncount": 0 "class": [10000, ], "intersectedquantity": , "intersectedcount": 1, "intersectionquantity": , "intersectioncount": 1 } ] } ], "intersectioncount": 1 #a categorical raster, will be summarized on unique values "layerid": 3, "method": "approximate", #approximate: area of interest represented as a grid, no area weighting. #interest is a polygon representation of grid, with area weighting. "intersectioncount": , "sourcepixelcount": , "intersectionquantity": , "pixelarea": , "geometrytype": "pixel", "values": [ "value": 1, "intersectioncount": 24090, "intersectionquantity": "value": 2, "intersectioncount": 38736, "intersectionquantity": "value": 3, "intersectioncount": 44753, "intersectionquantity": pr 1.3. Outputs 7

12 ] } "value": 4, "intersectioncount": 17088, "intersectionquantity": "value": 5, "intersectioncount": 129, "intersectionquantity": #a continuous raster, will only be summarized for intersection area "layerid": 5, "pixelarea": , "geometrytype": "pixel", "method": "approximate", "sourcepixelcount": , "intersectionquantity": , "intersectioncount": "layerid": 5, "pixelarea": , #area in hectares "classes": [ "class": [0,300], "intersectioncount": 67863, "intersectionquantity": "class": [300,310], "intersectioncount": 38677, "intersectionquantity": "class": [310,400], "intersectioncount": 18256, "intersectionquantity": } ], "geometrytype": "pixel", "method": "approximate", "sourcepixelcount": , "intersectionquantity": , "intersectioncount": "layerid": 5, "pixelarea": , "statistics": "STD": , "MAX": , "SUM": , "MIN": , "MEAN": "geometrytype": "pixel", 8 Chapter 1. Tabulate Tool

13 } "sourcepixelcount": , "intersectionquantity": , "method": "approximate" } ] } ], "sourcefeaturequantity": , #area or length of area interest, if polygon or "sourcefeaturecount": Error Handling This tool will almost always return successfully, because it is trapping and returning errors if encountered for each service and layer. These will be include the python stacktrace of the error to assist debugging, unless the error is an input or data error that the tool is specifically trying to handle. Additional information may be present in the logs to indicate the problem. Example error: } sourcefeaturequantity : , sourcegeometrytype : polygon, sourcefeaturecount : 1, services : [ layers : [ error : GCS_NOT_SUPPORTED: Geographic Transformation to WGS84 not found for projection layerid : 9 }], serviceid : u arcgis_geoprocessing_tools_test_data } ], linear_units : kilometers, area_units : hectares Supporting modules: 1.4. Error Handling 9

14 10 Chapter 1. Tabulate Tool

15 CHAPTER 2 Messaging 2.1 messaging.py class messaging.messagehandler(messages, logger=none) Convenience class for handling messaging. Includes a concept of progress, which is a measure of number of completed major and minor steps. For example, when processing a series of map services and their layers, the number of map services is the number of major steps, and the number of layers is the number of minor steps (represents loop-within-loop hierarchy). setmajorsteps(major_steps) Set the number of major steps to measure progress against. Parameters major_steps number of major steps of operation setminorsteps(minor_steps) OPTIONAL: Set the number of minor steps to measure progress against. Parameters minor_steps number of major steps of operation incrementmajorstep() Increment the current major step by one, and emit a new progress message. incrementminorstep() Increment the current minor step by one, and emit a new progress message. setprogress(progress) Emit a new progress message: PROGRESS [PERCENT_COMPLETE] Parameters progress the current progress, on a percent scale. addmessage(message) Emit a new message to both messages and logger (if available) Parameters message the message to emit 11

16 12 Chapter 2. Messaging

17 CHAPTER 3 Utilities A collection of utility functions and 3.1 PathUtils.py General path utilities for workspaces and routing to map documents behind map services utilities.pathutils.extractlayerpathfrommsdlayerxml(msd, xmlpath) Extracts layer data source from layer XML files stored in MSD. Parameters msd MSD file opened via ZipFile xmlpath path to XML file with layerinfo Returns list of layer paths, or None for each group layer; index in this list = layerid utilities.pathutils.getdatapathsforservice(serviceid) Extract paths for data layers in map service. Returns cached lookup if possible. Parameters serviceid Returns return list of layers paths (or None for group layers); order in this list = layerid 3.2 ProjectionUtilities.py General utilities for helping deal with projection related information utilities.projectionutilities.getgcs(spatialreference) Return the geographic coordinate system name for the spatial reference (e.g., GCS_North_American_1983). Parameters spatialreference ArcGIS spatial reference object utilities.projectionutilities.getprojunitfactors(spatialreference) Return the conversion factors for projection length to kilometers, projection area to hectares, and meters per projection unit for the spatial reference. Used to multiply the length and area calculated for features. Parameters spatialreference ArcGIS spatial reference object Note: Limited to projections based on Meter and Foot_US 13

18 utilities.projectionutilities.getwgs84geotransform(gcs) Find a general geographic transformation from one coordinate system to WGS These transformations are based on the continent scale transformations listed by ESRI, and are not exact for local calculations. Only the following source geographic coordinate systems are currently supported: NAD 1983 NAD 1983 HARN NAD 1983 CRS SAD 1969 ED 1950 Parameters gcs the input geographic coordinate system name extracted from the source projection well-known text Note: Target geographic coordinate system is always WGS 1984 utilities.projectionutilities.getgeotransform(srcsr, targetsr) Return the geographic transformation required to project between two projections (passing through WGS 1984, if required), or empty string if not required. Parameters srcsr source ArcGIS spatial reference object targetsr target ArcGIS spatial reference object Note: limited to the geographic coordinate systems supported by getwgs84geotransform utilities.projectionutilities.projectextent(extent, srcsr, targetsr) Project the extent to the target spatial reference, and return the projected extent. Creates a temporary feature class based on bounding box of source. Parameters extent source extent srcsr source ArcGIS spatial reference object targetsr target ArcGIS spatial reference object utilities.projectionutilities.createcustomalbers(extent) Given an extent in geographic coordinates, create a custom Albers projection centered over the extent that minimizes area distortions. Uses 1/6 inset from YMin and YMax to define latitude bounds, and centerline between XMin and XMax to define central meridian. Parameters extent extent in geographic coordinates Returns custom Albers spatial reference utilities.projectionutilities.getspatialreferencefromwkid(wkid) Returns a spatial reference object for WKID Parameters WKID ESRI Well Known ID Returns spatial reference object utilities.projectionutilities.isvalidareaprojection(spatialreference) Determines if projection is valid for area calculations. 14 Chapter 3. Utilities

19 Parameters spatialreference spatial reference object Returns True if valid for area projections, False otherwise 3.3 FeatureSetConverter.py Converts a featureset in JSON into a feature class in memory utilities.featuresetconverter.getfeaturegeometry(geomtype, geometry) Extract geometry from featureset JSON, and convert into geometry representation required for feature class. Parameters geomtype ArcGIS JSON geometry type: esrigeometrypoint, esrigeometrymultipoint, esrigeometrypolyline, esrigeometrypolygon geometry the geometry object extracted from JSON utilities.featuresetconverter.createfeatureclass(featureset, name= drawingfc ) Create an in-memory feature class from a featureset JSON. Parameters featureset the featureset JSON string. name name of output feature class (always in memory) Note: the original feature IDs (FID / OBJECTID) are not preserved in feature class, as they are built up fresh during construction of feature class FeatureSetConverter.py 15

20 16 Chapter 3. Utilities

21 CHAPTER 4 Requirements ArcGIS Server 10.2.x, including spatial analyst extension Published map services containing data to be operated against 17

22 18 Chapter 4. Requirements

23 CHAPTER 5 Installation - ArcGIS 10.2.x Note: This tool must be deployed to the root folder, with the name databasin_arcgis_geoprocessing_tools to properly work with Data Basin. 5.1 Pre-packaged versions These easiest way to install this tool is to deploy the latest service definition file from the downloads page Simply download, and then upload to your ArcGIS server. This version comes with a pointer back to the source Mercurial repository - develop branch (see below). 5.2 Packaging and deploying yourself Use the ArcGIS server command line tool ags_tool_deploy to help manage the deployment process. Install per the instructions in that repository. Download the latest development snapshot from develop branch or the latest stable version from master branch Then, from a command within the directory containing tools.pyt: <python_packages_dir>/arcgis_tool_deploy/deploy.py publish tools.pyt databasin_arcgis_geoprocessing_t use --overwrite if you want to delete and replace and existing service of the same name. Managing with Mercurial: Note: This assumes Mercurial is installed on your server. Given the active development and bugfixes on this tool, and the challenges in deploying to ArcGIS server, you can also clone this repository to your local machine using mercurial, and include basic repository information when you deploy the tool to ArcGIS server. This allows you to pull new updates directly to the ArcGIS server instead of having to redeploy the tool. Use the --hg option above to include Mercurial repository information. Then from within the installed location on the ArcGIS server reported using the publish command above, simply run hg pull --update to update to the latest changes in the branch you used above (make sure you are on develop for the latest changes or master for the latest stable changes. Once you have pulled and updated to the latest changes, simply restart the geoprocessing service. 19

24 20 Chapter 5. Installation - ArcGIS 10.2.x

25 CHAPTER 6 Testing Because this set of tools is built to run on ArcGIS server against running map services, it is necessary to execute the tests in the same environment. First, deploy this tool and make sure it is properly installed. Second, deploy the test data from the downloads page as a map service called arcgis_geoprocessing_tools_test_data Next, execute the tool at This will run the test suite as if it were a stand-alone geoprocessing tool. It will run through a variety of tests. 21

26 22 Chapter 6. Testing

27 CHAPTER 7 Known Limitations Tabulate tool is very slow due primarily to the implementation of the projection function in ArcGIS (arcpy). Work is underway to refactor out as many steps where tool is reprojecting data as is possible. A limited range of spatial projections are supported for target calculations and source map services, due to the ArcGIS requirement of including a geographic transformation to project between many different projections. Currently only continent-scale geographic transformations are included with these tools. Additional transformations can be added to utilities/projectionutilities.py Tool execution times vary with the number of layers, complexity of geometries, and extent of analysis. Expect analysis of several complex layers over larger areas to be slower. Path routing to ArcGIS layer data sources will not work for enterprise geodatabases. 23

28 24 Chapter 7. Known Limitations

29 CHAPTER 8 Authors Conservation Biology Institute developers: Mike Gough Brendan Ward Contact: databasinadmin at consbio dot org 25

30 26 Chapter 8. Authors

31 CHAPTER 9 License Copyright (c) 2014, Conservation Biology Institute All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. Neither the name of the Conservation Biology Institute nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS AS IS AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, IN- CIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSI- NESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CON- TRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAM- AGE. 27

32 28 Chapter 9. License

33 Python Module Index m messaging, 11 u utilities.featuresetconverter, 15 utilities.pathutils, 13 utilities.projectionutilities, 13 29