Environmental Modeling and Management using Free and Open Source Geospatial Tools

Transcription

1 Environmental Modeling and Management using Free and Open Source Geospatial Tools Ari Jolma Aalto University School of Engineering, Espoo, Finland, Daniel P. Ames Geospatial Software Lab, Idaho State University, Idaho Falls, ID, USA, Ned Horning Center for Biodiversity and Conservation, American Museum of Natural History, New York, NY, USA, Helena Mitasova North Carolina State University, Raleigh, NC, USA, Markus Neteler Fondazione Edmund Mach, GIS and Remote Sensing Unit. San Michele all'adige (TN), Italy. Aaron Racicot Ecotrust, Portland, OR, USA, Tim Sutton Centro de Referência em Informação Ambiental, Brazil, Summary to be written

2 1 Historical Background and Current Developments 1.1 Early Years The idea of sharing the source code of a computer program is probably as old as the idea of computer programming itself. For example Bell Labs distributed the early Unix distributions without charge and included the source code in the distribution (Raymond 2003). That way many universities and research organizations got their first operating systems and it also contributed to the later development of free Unix-like systems like GNU/Linux. Many of the early GIS were developed as unique applications or systems within the public domain and thus the commercial aspect of the software did not have a heavy influence. US federal agencies have always worked under the principle (which stems from the US copyright law) that anything produced by the federal government should not get a copyright. When the Corps of Engineers first developed the Geographic Resources Analysis Support System (GRASS) in the 1980's it was made freely available. When the development of the Internet made it possible to publish and download software, GRASS was used by many researchers around the world. Also many individuals made their work available on the Internet for others to use. For example Sol Katz at the U.S. Bureau of Land Management and Chris Evenden at the US Geologic Survey made significant contributions. In the early years environmental modeling, software was developed at research laboratories and at universities. Typically only the research results were published and sharing or publishing the software was not much considered. Some software was developed into proprietary products, while some was put into public domain. For example the source code of MODFLOW, which is a three-dimensional finite difference groundwater model first, has been freely available from the USGS since Typical early environmental models were non-interactive simulation programs written in FORTRAN which consumed and produced ASCII files. The idea of free and open source software (FOSS) was greatly advanced in the 1980's and 1990's with the introduction the GNU Manifesto 1 in 1985, the release of the Linux kernel 1.0 in 1993, and the subsequent development and widespread use of free Linux/GNU systems. However, during this period the geospatial and environmental software worlds developed more towards proprietary solutions. Many researchers and scientists collaborated with software companies and/or developed proprietary software while publishing their research in journals in writing. Both open source and proprietary software were also developed to have extension capabilities, which are used by many environmental modeling and management software developers who build own applications on top of that. Besides interfacing through data exchange, some software products (libraries mainly) became linkable to applications through the use of dedicated application programming interfaces (APIs). 1.2 Conferences on GIS and Environmental Modeling Environmental management, especially geophysical applications of analysis of digital elevation models, and assessment and planning of natural resources, have always been important application areas for geospatial software and sometimes the reason to develop such software. The 2- or 3- dimensional geophysical space is where environmental systems exist and thus it has always been important for the environmental modeling and management disciplines to store and analyse geospatial 1

3 data. Several influential conferences on GIS and environmental modeling were organized in 1990's. The First International Conference/Workshop on Integrating GIS and Environmental Modeling took place in Boulder, Colorado in 1991 and it was followed by a second conference in Breckedridge, Colorado 1993, a third conference in Santa Fe, New Mexico 1996, and a fourth conference in Banff, Alberta The papers from the first two conferences were published as edited books (Goodchild et al and 1996), as were selected papers from fourth conference (Clarke et al. 2002). Many presentations at these conferences presented results obtained with free software and one paper (van Deursen et al. 2000) explicitly addressed the issue of problems arising from the use of proprietary geospatial software for environmental modeling. While the conference series on GIS and Environmental Modeling was based in North America, two HydroGIS conferences were held in Europe. The papers from these conferences were published in the IAHS Series of Proceedings (Kovar and Nachtnebel 1993 and 1996). More recently, the biennial International Congress on Environmental Modelling and Software has been organized by a society with the same name (iemss). The forerunner for the iemss conferences is the MODSIM conference series, which has been organized in Australia or New Zealand since early 1990s. iemss 2010, which was held in Ottawa, Canada, was 5 th in its series, and it contained, besides standard sessions for scientific papers, workshops for discussing software licencing issues and use of open geospatial standards in environmental modeling. Also, a yearly FOSS4G conference has been organized by the Open Source Geospatial Foundation (OSGeo) since FOSS4G is mainly a conference for developers and users of geospatial FOSS but it has always had a significant attendance from the research community. The 5 th FOSS4G was held in Barcelona, Spain iemss defines itself as a not-for-profit organization uniting private persons and organizations dealing with environmental modelling, software and related topics. In practice it is a learned society linking researchers in the field. Geospatial FOSS and environmental modeling and management with free and/or interoperabile software is of interest within iemss. There have been several workshops and sessions focusing on software licencing, and geospatial software and its use in environmental modeling and management. A particular topic is open and collaborative environmental modeling à la open and collaborative software development. The approach is called community modeling, and a good example is the Chesapeake Community Modeling Program (CCMP) 2, which aims for integration of open source and institutionally supported models. 1.3 The Role of International Organizations The most distinctive feature of FOSS is that its development is a collaborative effort. The collaboration is open but managed by individuals or groups and aided by Internet-based tools making truly worldwide teams possible. Thus, it is natural that the collaboration will eventually develop into having a more formal organization. In the geospatial free and open source software this took place as the formation of the Open Source Geospatial Foundation (OSGeo) 2006 after the first Free/Libre and Open Source Software for Geoinformatics 2004 in Bangkok and a successful gathering of the tribes conference in Minneapolis, Minnesota The formation was catalysed by a proposal from Autodesk Inc. and a result of a lively discussion of the community. 2 ( )

4 OSGeo is an umbrella organization for various geospatial FOSS projects, which it incubates before they are admitted as OSGeo endorsed projects. Incubation includes software vetting for copyright problems and a health check for the related software community. Besides supporting and building open source software OSGeo also has aspirations regarding for example software interoperability standards; free and open geodata; study, use and development of geospatial FOSS in academia; and educating people about free geospatial software. These aspirations are often accomplished through collaboration with other organizations like OGC, OpenStreetMap, and University of Nottingham. OGC is the Open Geospatial Consortium, which develops interoperability standards for the geospatial data exchange and computations in the web. OpenStreetMap is an international movement to create and provide free geographic data such as street maps to anyone who wants them 3. University of Nottingham has a Centre for Geospatial Science (CGS), which hosts a Open Source Geospatial Lab (OSGL). The OSGL aims to be a hub for collaboration among various players in the field and become a place for academic research on and about geospatial FOSS. OSGeo has signed memorandum of understanding agreements with OGC, OpenStreetMap and CGS. The collaborative effort of people around the world to help during environmental disasters has become one of the hallmarks of the community developing and using free and open geospatial data and tools. The success of the OpenStreeMap community in helping deliver aid during the January 2010 earthquake in Haiti is a good example 4. More traditional and formal international organizations, exemplified by the United Nations (UN) family, have been much slower and not very pro-active in embracing the free and open source concept and technologies. A counter-example is GeoNetwork, whose development was supported by the Food and Agriculture organisation of the United Nations (FAO), the United Nations World Food Programme (WFP) and the United Nations Environmental Programme (UNEP) 5. GeoNetwork, which is a catalog application to manage spatially referenced resources, is now an OSGeo project. 1.4 Current Situation and Outlook Free and open source software has become an important part of computing. For developers a series of projects are existing to build on or join and platforms to start new projects. For users there are completely free operating systems available and hundreds of free applications, including scientific and engineering tools, that can be easily added to them from application repositories using graphical tools. There are free geospatial tools and tools that are useful in environmental modeling and management in these application repositories, but some important ones are still being added (for example Quantum GIS is not yet in the Ubuntu Software Center). Excellent examples of compendiums of geospatial FOSS exist, however. The OSGeo Live DVD 6 being the prime example. The GRASS community is perhaps the best example of a healthy software project that thrives on geospatial free software and where environmental modeling and management research is a prime target. In its more than 25 years of history, GRASS has evolved into a portable general-purpose GIS with outstanding functionality which has often been developed in academic projects and integrated into 3 ( ) 4 ( ) 5 ( ) 6 ( )

5 the core system. Some other similar cases exist, which are perhaps a bit more limited in their scope. 52 North is an industry-university collaboration, which studies and develops open source software, focusing on web-based systems for geospatial and especially the sensor web enablement (SWE is an OGC initiative). Other active open source geospatial communities include Quantum GIS, MapWindow, SAGA GIS, gvsig, GDAL and others. Many of these have strong links to environmental modelers and managers. 2 Fundamentals of Environmental Modeling and Management Environmental system are by their nature dynamic, complex, random, and evolving. The evolution, which may destroy, change, or create new systems, happens by human decision or by acts of nature. Natural evolution is a steady process or a sudden change or changes that often cause disasters from the human perspective. Environmental management, which is both adjusting to natural changes, and, increasingly, adjusting human decisions to be less harmful to natural systems, tackles local, regional, and global issues. Environmental problems have social, economic, and ecological dimensions and considerable inherent uncertainties. Environmental management should aim for sustainable use of natural resources, sustainable socio-economical conditions, and a functioning ecology. Environmental management is carried out with the help of observed data, prior knowledge of significant processes affecting the managed systems, institutions and methods of decision making, and communication. Models are increasingly used as aids in management. Modeling should be about systematic organization of data, assumptions, and knowledge for a specific purpose (Jakeman et al 2008). The complexity of environmental management means that the models need to integrate data and descriptions from multiple disciplines and that the results need to be communicated to various kinds of stakeholder groups. Environmental models include the following types. (i) Models of measured or observed data. The data may or may not originate from experiments. The data is organized into data structures. Numerous types of data structures have been developed for various purposes and goals, including representativeness of the observed environment, ease of access to individual data items, and suitability for specific algorithms. The metadata (context, descriptions and meaning) of the stored data is also a kind of a model. (ii) Descriptions of environmental systems. The descriptions may be qualitative, i.e., verbal or visual, or quantitative, i.e., mathematical. The descriptions can be hypothetical, based on prior knowledge of the laws of physics, human behavior, or other, or they can inferred from the data. The element that is described may be the internal structure of the environmental system, processes taking place within the system or at its boundaries, or the behavioral patterns of the system. (iii) Mathematical methods. There are several mathematical formalisms, which can be used for developing the quantitative descriptions further. Important formalisms include differential equations for describing change, statistical methods for developing models for random variables, various certainty calculi for inferring from relationships between variables. Each formalism comes along with several assumptions and mathematical methods for analyzing and computing results from the descriptions, (iv) Decision-making models. Action is an essential element in management. Alternatives for action need to come from somewhere, and there needs to be a procedure for evaluating the impacts of

6 different alternatives before a decision is made. A fundamental model for decision making is a provided by rational choice theory, which, however, still leaves much room for modeling especially in the case of incomplete knowledge. Environmental management has a geospatial dimension by nature and often the most important questions are about what happens in one place when something is done in another place. Geospatial information, i.e., information that is tied to a location and information about how locations are connected to each other, is thus essential for environmental management. The exactness of location information is often overshadowed in environmental modeling by the need to be able to link locations to other locations in a schematic fashion and by the need to know about changes (over time) in the selected location. Environmental models are often conceptual by nature: the conceptualized elements refer to more or less hypothetical objects or storages (for example depression storage of water) in the geospatial reality. There is a long-standing discussion in the environmental modeling literature about the real benefits of increasing spatial accuracy and explicitness of environmental simulation models (see for example Abbot and Refsgaard 1996). Environmental models are often integrated together and with other tools to form environmental decision support systems (DSS). Environmental DSS can be divided into three categories (in order of specificity) (Rizzoli and Young 1997): (i) Generic environmental DSS is a toolkit that provides functionality for managing and exploiting data (including spatial data), knowledge (including expert knowledge), models, and solution methods for diagnosis, planning, management, and optimization. (ii) Problem specific environmental DSS is similar to a generic environmental DSS, but focuses on a specific problem, for example the effects of runoff in urban areas. A problem specific environmental DSS needs a clearly defined interface to a data base. (iii) Situation and problem specific environmental DSS is similar to a problem specific environmental DSS, but is built for a specific problem and for a specific location. The developer has three main tools and techniques at hand for delivering a usable DSS: programming languages, modeling tools, and model integration tools (Rizzoli and Young 1997). An important goal in environmental management is integrated assessment 7 (IA). Model coupling (see below) is one method to develop tools for IA. There are also specific modeling approaches, which have characteristics that make them suitable for IA. One class of IA models utilize conceptually very highlevel variables combined with a technique such as system dynamics 8 or Bayesian networks (Pearl 1988). These types of models are typically developed top-down, focusing on the perceived problem. Another kind of IA models are developed bottom-up, focusing on the interactions of a large number of autonomous agents. The space is external to the top-down models the variables may be linked to locations or spatial scenarios while the bottom-up models often explicitly require an artifical spatial world for the agents. 2.1 The Modeling Process 7 ( ) 8 ( )

7 Modeling proceeds in a step-by-step fashion, with frequent assessment of the validity of the choices made earlier in the process. Modeling, as any activity, is usually limited by time and other resources. Modeling is also typically affected by prior knowledge, experience, and preferences of the modeler or the modeling team. In particular the preferred mathematical formalism has often a strong impact on the modeling carried out. Development of integrated environmental models and DSS is typically problematic due to difficulties in communication across disciplines. Jakeman et al (2006) present a synthesis of the development and evaluation of environmental models. The steps a modeler should take in modeling are according to them the following. (i) Define model purpose. The modeler should know what the model will be used for. (ii) Specify modeling context. The modeling context contains the information and social environment in which the model will be used and the requirements set by those. (iii) Conceptualize the system being modeled, specify data and other prior knowledge. The essential internal structure of the model should be drafted. (iv) Select model features: nature, family, form or uncertainty specification. The mathematical machinery to be exploited as well as the spatial and temporal discretization is selected. (v) Determine how model structure and parameter values are to be found. The amount of observed data and prior knowledge there is of the system determines the detailed characteristics of the model. The rule of parsimony should be obeyed, i.e., there should be no unnecessary complexity in the model. (vi) Choose estimation/performance criteria and algorithm. There are several methods to choose from and problems may arise with how to treat integrated models. (vii) Identify model structure and parameter values. There may be more than one possibility for the model structure and parameterization, which need to be evaluated side by side. (viii) Verification including diagnostic testing. It is important that all existing observation data is not consumed in the previous steps so that some is left for independent verification. This step should also contain sensitivity analysis of the model and building of appropriate trust with the model. (ix) Quantification of uncertainty. Before the model is used, its capability to cope with uncertainties in data, measurements, or baseline conditions need to be well understood. (x) Model evaluation or testing. The model is used for its intended purpose while observing its fitness and possibly comparing it with other models. The geospatial domain is typically information rich and spatially detailed. Thus it may be tempting to use all the available data as such and downscale knowledge of processes to the same level of detail. This may be warranted, but in any case the decision should be a conscious one. There are also many geo-analytical methods for identifying spatial units, i.e., areas that do not exhibit too much spatial heterogeneity to be treated as units in the modeling (see for example DeMers 2002 for the raster case and O'Sullivan and Unwin 2003 for the vector case). Environmental models can be based on homogeneous or spatially averaged units or they can be based on fully distributed representation by

8 continuous fields. For example, in hydrologic applications, the averaged units represent watershed hierarchies, hillslope segments, channels and stream networks (Mitasova and Mitas 2001). This approach is very effective for systems which include structures (urban hydrology, agricultural fields), however, adequate selection of units can require substantial expertise for complex, natural environments. Distributed models rely on different discretizations of fields/multivariate functions, such as regular or irregular grids or meshes, derived from the numerical methods used for solving the governing equations (finite differences, finite elements or path sampling). 2.1 Software for Modeling The software for modeling includes all kinds of data processing tools, tools for processing and implementing descriptions, and tools for using the models to find answers to questions. The relationship between (the coupling of) geospatial software and modeling software has been discussed a lot in the literature (see for example Fedra 1996, Mitasova and Mitas 2001, and Brimicombe 2009). There are several possibilities for this relationship, and deciding which one to select depends on both technological and modeling aspects. The technological considerations relate to possibilities for how to arrange data exchange between geospatial software and model software, and to possibilities for programming with the geospatial software. The modeling aspects relate to the spatial nature of the model, and the need for geo-analytical methods in the modeling. Several possibilities for coupling have been identified. (i) Loose coupling means that information exchange between the geospatial software and the model software is by data storage. Depending on software there may not be a shared data storage and the coupling is further complicated by format conversions, which may lead to data loss. Closed data formats make loose coupling especially troublesome. Loose coupling complicates the modeling process but it may be justified because of its simplicity, division of labor, or other reasons. (ii) Tight coupling means that data is exchanged between geospatial and model software without storage in between. Tight coupling can be achieved by some form of inter-process communication (IPC) or simply by linking the software. In the latter method one software provides the user interface and the other provides its functionality through an application programming interface (API). There are several methods and implementations of IPC for various operating systems. Some IPC methods are designed for computer networks. (iii) Embedding modeling support into geospatial software or geospatial support into modeling software is a form of coupling where the distinction between modeling and geospatial cannot be made any more. The need and/or requirement for embedding is that the model needs to be inherently spatial and that the basic computational framework is extended with geospatial and modeling support. A popular fundamental methodology for developing explicitly spatial models is map algebra. For example DeMers (2002) presents a treatise on modeling using map algebra. In general, geospatial software is strong, in supporting spatial data and cartographic projections, and in supporting visual and interactive exploration and management of the data. Geospatial software also typically support a large set of common data transformation and geo-analytical methods that may be of value in the modeling project.

9 3 The Geospatial Free and Open Source Software Toolchain A software toolchain is a set of tools that are used to carry out a workflow. A workflow is a sequence of a connected tasks that need to be completed in order to carry out a task, for example software development or modeling. Typically the tools of a toolchain are in some way interoperable, have a similar look and feel, use the same programming language, or have something else that makes them attractive for a user to use together. A geographic information system (GIS) a loose term in itself is a set of tools and thus a toolchain may be implemented within a GIS. A software platform is technology on or for which a solution is developed. We separate between a development platform, which consists of a software development toolchain, and a delivery platform, which is a platform a solution will be used on. A development platform may be tied to a delivery platform but it can also be rather independent due to standards or other reasons. Software platforms guide usage of software, may dictate organizational purchases, and direct developers' interests. Activities, which increase interoperability, such as development of standards, that increase possibilities for coupling software, and porting of tools to new platforms, make cross-platform or multi-platform solutions possible. Again, a (specific) GIS can be a platform, as solutions often can be developed and delivered for a (specific) GIS. But, as standardisation proceeds, the same solution may possibly be easily ported to another platform. A software stack is a set of software developed in such a way that software higher up in the stack exploits functionalities provided by software lower in the stack. The dependencies chain of a free and open source software (FOSS) package may be several layers deep and it may be split up into several threads. There are thousands of packages (Synaptic Package Manager in Ubuntu lists packages as of this writing in late 2010) and as each software package is often developed more or less continuously and independently it is easy to understand that management of distributions of systems based on free software is an extremely complex task. The world of free software and the world of proprietary software are not separate. All kinds of mixtures exist. For example the GNU software development toolchain has been ported to the Microsoft Windows operating system platform, free software may be developed using proprietary tools (for example the Linux kernel development used for a long time a proprietary versioning system), free software stacks can be built and delivered for proprietary platforms, and solutions can be built on platforms and using stacks that mix free and proprietary software. 3.1 Hardware and Operating System Platforms The most common operating systems are those that belong to the Microsoft Windows family and those that are Unix or Unix-like (e.g., Linux). An operating system manages the computer hardware and provides services for application software. The distinction between operating system software and application software is not sharp, however. Hardware platforms can be divided into servers and workstations or personal computers, and personal computers can be further divided into desktop and mobile computers. This distinction is not sharp, however, but mainly reflects the main usage mode. There are both Windows and Unix systems in all categories. The mobile platform is currently developing very fast and due to this and other reasons it is a rather difficult platform to develop free software for.

10 In environmental modeling and management it is important to work both in the field and in the office and utilize baseline data and centralized information services. Thus all hardware platforms are important for the sharing and exchange of data between systems and applications. Geospatial FOSS libraries are in principle available for all major platforms, but applications need special considerations. An example of a geospatial FOSS application for a mobile platform is gvsig Mini, which is mainly a viewer for geospatial content. For workstations there are several geospatial FOSS applications, for example GRASS and Quantum GIS. The main server applications are MapServer, GeoServer and GeoNetwork, which are for publishing geospatial data and metadata. 3.2 Programming Language Specific Platforms Geospatial tools contain executable programs written in programming languages but also programming languages themselves are important tools in modeling. FOSS code is made available freely and openly, but it is not usable without a platform that supports it, and, further, the code needs to be compiled and linked into executable programs for the chosen platform. There are practical limitations to using more than one programming language together in one solution. Developer toolchains tend to support only one programming language at a time, many programming languages require specific components in the platforms, and linking software written in diverse languages into working stacks has specific requirements Systems Languages Systems languages are programming languages that are used for the operating system and for the bulk of the applications. Typical systems languages are C and C++. The central geospatial FOSS libraries written in systems languages are Proj4, GEOS, and GDAL/OGR, but there are others like Mapnik, which is a library for rendering maps, and CGAL, which is a library of computational geometry algorithms. Proj4 is cartographic projections library, GEOS is a library for modelling and manipulating 2-dimensional linear geometry objects according to the Open Geospatial Consortium (OGC) Simple Features Specification, and GDAL/OGR (GDAL for short) is a library for geospatial data (the core of GDAL is a raster library and OGR is a vector library). The systems language libraries form a software stack. GDAL uses (in fact, may use, as that is a decision made when the source code is configured for compilation) Proj4, GEOS, and numerous other libraries for specific tasks and data formats. Mapnik uses GDAL for data access (it has also its own plug-in architecture for data sources). GDAL has many uses. First, it is a translator library for data formats as it supports several raster and vector data formats though a plug-in architecture. Second, it can be used for reprojecting data sets, and third, it provides a platform for geospatial algorithms. GDAL source code contains implementations of several commonly needed geospatial algorithms, including contouring raster data, rasterizing vector data, gridding point data, and others. The list of available algorithms is extended with algorithms written in Python Java The Java platform is an important FOSS platform, which can be used on all three main platforms (mobile, server, workstation) and which comprises a large set of tools for general computing. There is a

11 large collection of Java geospatial FOSS, which includes libraries, servers and applications. GeoTools is a free Java toolkit, which contains the Java Topology Suite (JTS), support for coordinate reference systems and transformations, data access through several data format drivers, graphs and networks, symbology, and rendering High Level Programming Languages A high-level programming language may be a general purpose language, like Perl and Python, or it may be primarily intended for a specific purpose, like Octave (numerical computations) or R (statistical computing and graphics). High-level languages are generally intended to be easy to use and for practical problem solving. Many high-level languages (including Perl, Python, Octave and R) are FOSS. Established high-level languages often have a large library or module base for various common tasks. The Perl module repository is Comprehensive Perl Archive Network (CPAN), the Python Package Index (PyPI) is a repository of Python software, Octave-Forge is a central location for the Octave packages, and Comprehensive R Archive Network (CRAN) is a repository for R code and documentation. In order to provide capabilities for geospatial computing a high-level language should support spatial data, spatial objects, cartographic projections, spatial algorithms, and/or suitable graphics. Sometimes these functionalities are/can be developed into the language itself but often the languages are extended with geospatial libraries written in systems languages using a mechanism called foreign function interface (FFI). A FFI (also sometimes called bindings ) is typically code, which allows calling subroutines from a library and takes care of conversions between data representations. Often FFIs employ specialized interface languages and tools for the binding workflow. FFIs may be greatly complicated due to differencies between the languages. For example dynamic languages like Perl and Python contain memory managers while systems languages like C and C++ require programmers to explicitly allocate and free memory. This is not always handled by the bindings. For example Python code written for Python GDAL bindings need to follow certain rules in order to avoid fatal errors. Dynamic languages also have very different type systems from systems languages. For example the simplest data type in Perl is a scalar, which is literally one thing and only internally converted to an integer, a floating point number or a string as needed. A FFI may implement callbacks, subroutines written in high-level language that may be called from the systems language. This is useful, when the systems language subroutine may take a long time and need to report information about its progress to the user. On the other hand, a call-back technology is essential for implementing high-level language plug-ins for applications written in systems languages. For example Quantum GIS, which is written in C++, can be extended with plug-ins written in Python (although the architecture is still considered experimental as of this writing). SWIG is a FOSS tool for creating FFIs for several high-level languages to libraries written in C or C++. There are SWIG interfaces to GEOS (Python, Ruby), GDAL/OGR (Python, Perl, C#, R, Ruby, Java and other), and MapServer (Python, PHP, C#). R is a language and environment for statistical computing and graphics. R is an implementation of S, which is a popular data analysis, visualization, and manipulation language. There are several packages for R for spatial statistics. The Spatial Data Task View available on the CRAN lists over 90 packages that can be used for reading, vizualising, and analysing spatial data. For example, the sp package

12 provides classes and methods for spatial data in R. Many other R packages depend on sp, including rgdal, which provides bindings to GDAL. Further bindings exist to GRASS, SAGA and Quantum GIS. Octave is a high-level language primarily intended for numerical computations. The Octave language is mostly compatible with the MATLAB language. MATLAB has its roots in University of New Mexico, but is since 1984 controlled by a company (MathWorks), which develops the MATLAB application. MATLAB application has a mapping toolbox, but the respective Octave mapping toolbox is still in its infancy. Although there has been discussion about linking GDAL to Octave (Niles 2006) this seems to not yet be the case. Perl and Python have also a broad range of packages available that make them suitable to scientific computing (i.e., modeling). Suitable Perl packages in this respect are for example Graph (for graph operations) and PDL (for numerical computations with N-dimensional data arrays). For Python there is Numpy, which provides a multidimensional array class and associated operations. Both PDL and Numpy extend the respective languages with efficient data storage and fast routines implemented in C Process Virtual Machines A process virtual machine (VM) is an application that provides an operating system independent programming environment. Some process VMs support only one programming language but some support several programming languages. Thus process VMs may support also programming language interoperability. In some cases there are more than one implementation of a particular process VM specifications. The java VM is a crucial component of the Java platform. There are several implementations of the java VM for various platforms. Java VM supports mainly the Java programming language 9. The Common Language Infrastructure (CLI) is a standardised specification for a process VM, which forms a core component of the Microsoft.NET framework. The.NET Common Language Runtime, which is an implementation of the CLI, supports several programming languages. There is also a FOSS implementation of the CLI developed within the Mono project. The use of the CLI, although an open standard, is questioned by some FOSS developers because the specification is partly covered by patents owned by Microsoft and other companies. The FOSS desktop GIS MapWindow relies on the.net framework. Parrot is a fully FOSS virtual machine that aims to support several dynamic programming languages like Perl and Python. Parrot has been developed since early 2000's and the version 1.0 was released in 2009, but the support for many real programming languages is still only partial. Thus the real impact of the Parrot VM remains to be seen. 3.4 Application Development Platforms Development of a problem and location specific application is often an important part of an environmental modeling or management project. Information system and application development typically follow a life cycle of analysis, design, implementation, and maintenance. During the analysis 9 ( ) is a list of programming languages for the Java VM.

13 phase the existing situation is studied to understand what the problem is and what are the requirements for solving it. During the design phase developers engineer a workflow or specifications for an application or an information system that fulfill the requirements and solve the problem. Implementation is a technological phase, where the workflow is mapped to a concrete toolchain, which is made to work together, or a system is put together (this may involve writing new code) according to the specifications. Through regular maintenance the tools and the system are kept up-to-date and working. Several methodologies exist for working through this life cycle or a refined version of it 10. Environmental modeling and management consist of several more or less separate activities and for each activity there are several possible tools and toolchains. An activity in the modeling workflow could be for example calibration, validation, testing, or running simulations with a model. An activity in environmental management could be for example assessment of certain environmental resource or developing a policy for protecting it. There is no single rule for solving the problems and carrying out the required activities. Some people may try to find the best tool for each task, while some try to use the single tool they know best. Below we go through a list of mainly technical task categories and present a stack of mainly geospatial FOSS tools for solving problems and fulfilling requirements. (i) Data management Data management is a critical function of geospatial environmental applications. Both the number of required geospatial datasets and their size can be voluminous. Geospatial datasets are stored within specialized file formats or in databases using either a vector or raster based data storage model. Geospatial data sets are often provided for use from servers either through networked file systems or via the Internet. Descriptions and metadata of the data sets are usually stored and managed along with the data. Increasingly, the central solution for data storage is the use of relational databases and associated management systems (RDBMS). Due to the richness of associated tools, especially the structured query language (SQL), RDBMS, with client applications and other tools, may be used as a platform for developing applications, functionality, and/or services. The SQL can be used for both managing the table and index structure for the data and for managing and querying the data itself. On the FOSS platform two main RDBMSs are MySQL and PostgreSQL. A general perception is that MySQL is less featured (implements less of the SQL standard including the spatial SQL) than PostgreSQL but it is faster. Thus MySQL is probably more popular in Web-based solutions and PostgreSQL in desktop solutions. Geospatial data are not easy to store in a standard RDBMS, thus spatial extensions have been developed and standardised. PostGIS is an add-on to PostgreSQL, which provides the standard spatial data types and spatial queries. Spatial data handling capabilities have also been added to MySQL recently. A more recent and lightweight system is SQLite which is a file based DBMS which does not require a server. The SpatiaLite extension ( enables SQLite to support spatial data in an OCG conformant way. Also raster data is sometimes required to be stored in a RDBMS although this issue has been much discussed and the benefits questioned also in the OSGeo community (see for example this thread 11 ). There is an ongoing project to include raster support into PostGIS 12, additionally it is supported in SpatiaLite ( ) provides one list 11 ( ) 12 ( )

14 An alternative for data management in a RDBMS is to store it as files in a filesystem. Mixed solutions may use RDBMS to store metadata and paths to data. (ii) Data integration Environmental modeling often requires integration of various geospatial data from multiple sources. An important component of the geospatial software stack is a tool to transform datasets from one geospatial coordinate system to another. Solutions include creating an application/project specific data base, where all data is required to be in a single coordinate system, or depending on online/on-the-fly data transformation functionalities of the applications. Other data integration tasks are homogenisation of raster data attribute values and color tables; homogenisation of vector data regarding attributes; and conversions of vector data from linestrings to polygons etc. In general the GDAL suite of tools and GeoTools in the Java platform contain all the necessary tools for these tasks. (iii) Data serving Data serving is a functionality that is required for loose coupling or standards-based forms of tight coupling of subsystems as well as for applications that publish data. Commonly used data servers include RDBMS and web servers that serve geospatial data. OGC has published standards for serving maps as images (web mapping service, WMS), raster data (web coverage service, WCS), and vector features (vector feature service, WFS). MapServer and GeoServer are the two main FOSS tools for geospatial web services. They are development environments for building spatially-enabled Internet applications, supporting various OCS standards. Mention WPS? Deegree, 52N and ZOO ( (iv) Geo-analytical tools Geo-analytical tools include numerical calculation, geostatistical, and computational geometry operations. The basic analytical method family for raster datasets is map algebra (a term coined by Tomlin, Tomlin 1990), which extends the standard algebra of scalar values to raster data. Map algebra extended with methods for hydrological analysis of raster digital elevation models (DEMs) is directly usable in environmental modeling. For example, raster-based spatial analysis provides the most mature, core GRASS GIS functionality (Fig. X1). Besides the standard tools, such as spatial queries, profiles, basic statistics, buffers, and neighborhood operators, it includes powerful map algebra module. Cost surfaces, and least cost path modules support various optimization tasks while line of sight module can be used to provide input for location of communication towers, wind turbines, emergency management or assessment of impact of high buildings. Shading, sun illumination and computation of solar energy maps provide important inputs for ecosystem modeling and assessment of photovoltaic energy potential. Comprehensive set of tools for terrain modeling, flow routing, and watershed analysis allows users to compute terrain parameters and hydrologic features that are essential for environmental modeling and management. Basic support for 3D raster data (voxel volumes) including 3D map algebra, 3D volumes interpolation and 3D groundwater flow modeling are also included.

15 FIGURE X1 here - raster analysis Figure X GRASS GIS provides tools for complex geospatial analyses, including cost surfaces: (a) cumulative cost surface representing travel time from any point within the study area to the closest highway, displayed as a surface in perspective view; (b) locations within the cost distance less than given threshold displayed as a 2D image along with a complete street network. Geospatial analysis based on vector data is also supported by the FOSS4G software. For example, GRASS GIS provides tolls for vector network analysis such as the shortest path, traveling salesman (round trip), allocation of sources (subnetworks), minimum Steiner trees (star-like connections), and iso-distances. (v) Geovisualization In environmental modeling and management information about the geospatial reality is essential. Geographic visualization (geovisualization) includes methods and tools for presenting geospatial information. The aim for the method or tool may be either communicating existing knowledge or knowledge construction. The mode of the latter is exploratory and it is usually facilitated by visual interactive interfaces (Thomas and Cook 2005). Typical features of geospatial software that are able to render images are: * associate display colors with attribute values (for example a lake feature is rendered blue) * use symbols for point data (for example measurement points are marked with symbol depicting a sensor) * use line widths, hatching and other similar techniques for vector data (for example land cover types can be portrayed with specific hatching) * add labels to features (for example measured values can be shown for illustration) * show several data sets at the same time controlling the drawing order and transparency (for example plans can be shown on top of aerial photographs) * create perspective 3D images (for example a bird's-eye view of a mountaineous site greatly enhances the understanding of the geophysics of the site) Typically features that are missing or not well-developed but would be useful in environmental modeling and management are the ability to:

16 note - if this is an electronic book we can actually include some animations here * show spatio-temporal data sets using animation or other techniques (for example changes in water quality of a lake can be depicted with simple animation involving changing colors), although 2D and 3D animations of environmental data and modeling results have been well supported in GRASS GIS (Mitas et al. 1995, Mitasova et al. 2010) * link schematic (model) view to geospatial (map) view (for example a water supply system may be easier to understand if geographical distances are removed) * allow sketching of designs and plans over maps (this kind of functionality would be useful for example during interactive design sessions with stakeholders). A related, exploratory 3D analogy of sketching was developed with GRASS GIS as a Tangible Geospatial Modeling System (Tateosian et al. 2010), Figure XX Figure XX Tangible Geospatial Modeling System coupled with GRASS GIS allows users manually modify landscape model and explore its impact on runoff or flooding: (a) dams or buildings can be added (b) flow simulation on initial model and on modified landscape, (c) exploring impact os a small breach in protective dune on barrier island flooding. The mainstram geospatial FOSS (as also to a large extent the proprietary GIS) focuses on producing decent maps automatically from geospatial data, while software for more explorative geovisualization remains mostly university research projects. GRASS, having strong ties to the academia, has some powerful geovisualization tools (see below). Well-known university-based software for explorative geovisualization include CommonGIS (Andrienko and Andrienko 2004), LandSerf (Wood 2009), and GeoVISTA Studio (Gahegan et al 2007). CommonGIS is a system for cartographic visualisation and non-spatial graphs, and a toolkit for querying, search, classification, and computation-enhanced visualization of geospatial data. It is freely available in binary format (thus, no source code) for non-commercial use. Landserf is a system for visualization and analysis of geographic surfaces. The source code of Landserf is freely available but the copyright is restrictive (thus, not really FOSS). GeoVISTA Studio is a programming-free

17 environment that allows users to quickly build applications for geocomputation and geographic visualization. GeoVISTA is licenced under LGPL (thus, FOSS however the download link on the homepage leads to a Java Web Start link instead of source code download). Notable geovisualization FOSS considering environmental modeling and management includes vtlib, Epigrass ( and nviz and xganim within GRASS. Vtlib is the main product of the Virtual Terrain Project, whose goal is to foster the creation of tools for easily constructing any part of the real world in interactive, 3D digital form. Epigrass is software for visualizing, analyzing and simulating epidemic processes on geo-referenced networks. Although Epigrass is aimed for epidemic processes, it can be hypothesized that it can be applied for certain environmental processes also because it is FOSS. Nviz (Fig XXX)is a 3D visualization and animation tool and xganim is a tool for animating raster timeseries. GRASS has furthermore been linked to Paraview ( and The Persistence of Vision Raytracer (POV-Ray, FIG XXX Advanced visualization in GRASS supports interactive adjustment of lighting: application to analysis of flow patterns over landslide areas extracted from the first return LiDAR point cloud. The rough surface represents forested areas. Interactive 3D environments, which can be useful for for example environmental scenarios, can be created with many FOSS libraries developed within the computer game community. (vi) User interfaces The user interface of an application permits the user to manipulate the behavior of the application and for the application to display information to the user. The reason for the manipulation may be one of three things (interpreting the Jackson's (2001) basic problem frames): controlling the way information about the real world is presented, allowing the user to edit a work piece (data structure) maintained by the application, or controlling the behavior of a system (or its model). In a typical application the user interface serves all of these purposes but it is useful to distinguish between these three fundamentally different problems.