ArcGIS Data Models: Raster Data Models Jason Willison, Simon Woo, Qian Liu (Team Raster, ESRI Software Products)
Overview of Session Raster Data Model Context Example Raster Data Models Important Raster GDB Considerations Rasters in ArcGIS 9.0 Questions / Answers / Comments Sample Database and Design Documents For more information, go to http://support.esri.com/datamodels
What is a Data Model? A practical template for implementing GIS solutions Built on consensus within a discipline Both providers and users Goal is consistency across organizations Great starting points Designed to adhere to emerging standards
Why Data Models? Data models provide Quick Start" solutions Best practices Optimized performance Improves data sharing More successful GIS implementations The ArcGIS geodatabase architecture allows Ready-to-use frameworks Centralized location Intelligence/Rules
Raster Data Model Context Raster Data Models Essential parts of other data models Just another layer in your data model stacks Base-mapping and/or Analysis needs ESRI s Geodatabase Raster Solution Enterprise or personal geodatabase We will focus on enterprise (ArcSDE) Easy to build specifically to meet your needs
Raster Data Model Context
Demonstration
Where do I start? Determine The purpose of my raster data Your application s scale of use The properties of the raster data All this contributes to your database design Use these guidelines to optimize your solution
Types of Geodatabase Rasters Raster Dataset Storage of a single raster dataset Mosaic many inputs to one continuous raster dataset Raster Catalog Container of independent raster datasets Organized as one entity in the database Raster Attributes of a Feature Class Feature class attribute Stored in the geodatabase
Raster Datasets Input files Geodatabase Raster Dataset 3712a.tif 3712b.tif UC2004.IMAGE_3712 3712c.tif 3712d.tif Single geodatabase raster dataset created from multiple data sources Single row table in the geodatabase Overlapping pixels have options (overwrite, ignore, blend, min, max) Use for DEMs, most base-mapping, and where fast viewing at any scale is important
Raster Catalogs Collection of Raster Datasets stored in one table Designed for Overlapping data (time series, stereo-pair photos, scanned maps) Situations where storing each dataset independently is important When you want to organize similar data Use entire catalog, or a subset ArcSDE Raster column UC2004.AERIAL_CATALOG IMAGE NAME DATE 1 WC_4724 05/26/98 2 WC_4725 07/05/01 3 WC_4726 10/04/01 4 WC_4727 02/26/02
Raster Attributes Geodatabase raster data is associated with geodatabase features
Personal Geodatabases Raster Data in the personal geodatabase Pixels are not stored inside Access Raster Data is stored as referenced files Managed Raster Data is copied as IMGs to a special folder **Note: Raster Datasets are always managed Unmanaged Raster Data is simply referenced by a pathname **Note: Raster Catalogs and attributes can be managed or unmanaged Demonstration
Enterprise GDB, Personal GDB, or Raster Files Enterprise GDB Our best multi-user solution Large data volumes (terabytes) Multi-platform / multi-database Personal GDB Good for prototyping and small implementations Limited number of connections Small quantity of data (gigabytes) Windows only
Storage / Hardware Two local servers (here at UC2004) 100 megabit network DELL PC Dual 933 MHz processors 1GB of RAM 1TB normal disk storage IBM PC Dual 3 GHz processors 4GB of RAM connected to an INLINE storage device 2 disk arrays, 1.6TB each
Raster Data Model Case Studies Ortho-rectified Imagery Global Raster Raster Elevation Scanned Map Time Series Raster Attributes
Ortho-rectified Imagery Topography Data Model Cartography Administrative areas Parcels Uses, rights/interests, and ownership Legal description Corners and boundaries Survey control Vectors (roads, rail, buildings) Ortho-rectified Imagery Very accurate imagery
Ortho-rectified Imagery Hydrography Data Model Streams Hydrographic points Drainage areas Vector water features Channels Surface Terrain/Elevation Hydro response Ortho-rectified Imagery
Ortho-rectified Imagery Data Model (Example Implementation) Texas Natural Resources Information System (TNRIS) Responsible for managing data for all of Texas 1 meter Color-Infrared Digital Ortho-rectified Quarter Quads (DOQQs) Used a Raster Catalog ~ 440 GB of raw data 3000 of 16000 provided to ESRI Demonstration
TNRIS Ortho-imagery Raster Data Model Purpose: Base-Mapping Update schedule: Irregularly scheduled updates Has portions of the state irregularly updated each year by the counties of Texas (** Not part of the decision process anymore in 9.0 **)
TNRIS Ortho-imagery Raster Data Model Scale of Use: Seems a bit too conservative for monitors, OK for printing (** assumes user will not use lossy compression **) (Table courtesy of EMERGE)
Orthorectified Imagery Scale of Use Considering Lossy Compression M A P S C A L E 1 : 1000 1 : 5000 1 : 12000 1 : 24000 1 : 63360 1 : 150000 1 : 250000 more loss JPEG Compression less loss 6 15 25 50 75 94
TNRIS Ortho-imagery Raster Data Model Database Design: Enterprise Geodatabase Size of the data Personal would not have scaled enough Raster Catalog To preserve their base-mapping unit To preserve overlapping corners Lossy compression (JPEG) Base-mapping (not for analysis) Cubic Convolution pyramid resampling Continuous data
TNRIS Ortho-imagery Raster Data Model Data Properties: Source Data: 440 GB total raster input data 3000 TIFF files (150 MB each) Enterprise Geodatabase Data: 135 GB in the Geodatabase Raster Catalog ArcGIS Layer: Use scale dependency to view Primarily 10 to 20 rasters at a time Up to 1 : 3780 (1:1 screen to pixel in the monitor) Printing would be different, inline with industry recommendations
L.A. County Ortho-imagery Data Model Very similar to TNRIS Base-mapping Lossy compression (JPEG) Differences 0.6 meter resolution Natural color Bilinear Interpolation pyramids Required fast display at any scale Raster Dataset Demonstration
Raster Data Model Case Studies Ortho-rectified Imagery Global Raster Raster Elevation Scanned Map Time Series Raster Attributes
Global Raster (Example Implementation) EarthSat s SNC of the world Contains all land masses 15m resolution Approximately 4.5 TB of data ~3600 TIFF files (all >1GB) Used 9 independent Raster Datasets Mosaicked a portion of their files into each Demonstration
Global Raster Very similar to TNRIS Base-mapping Lossy compression (JPEG) 1 meter resolution (Natural Color) Cubic Convolution pyramids Required fast display at any scale Also required fast loading period 9 Raster Datasets 9 clients loading data into 1 server < 1 week to load into the geodatabase
Raster Data Model Case Studies Ortho-rectified Imagery Global Raster Raster Elevation Scanned Map Time Series Raster Attributes
Raster Elevation Data Hydrography Data Model Streams Hydrographic points Drainage areas Vector water features Channels Surface Terrain/Elevation Hydro response Orthorectified Imagery
Raster Elevation Data Model (Example Implementation) Southern California Association of Governments (SCAG) 5 meter raster elevation dataset L.A. County provided by Emerge/Intermap Demo
Raster Elevation Data Model Purpose: Derive additional datasets (analysis) Update Schedule: Very infrequent updates (** Not part of the decision process anymore in 9.0**)
Raster Elevation Data Model (Table courtesy of USGS) Scales of Use: Because these are interpolated values, these Scales set by the USGS are valid thresholds. Based on this information, we can infer that 5m data is valid for mapping up to about 1:4000.
Raster Elevation Data Model Database Design: Raster Dataset Mosaic together Overlap is the same surface, so it does not need to be preserved Lossless compression (LZ77) Data will be derived (analysis) Cubic Convolution Pyramid resampling Continuous data
Raster Elevation Data Model Data Properties: Source Data 2.25 GB, 2 GRIDs 5 meter Intermap DSM (surface) RADAR product Geodatabase Data One large raster of 3 GB Lossless LZ77 ArcGIS Layer Use at any scale (up to 1:4000) Derive other layers as needed (slope, aspect)
Raster Data Model Case Studies Ortho-rectified Imagery Global Raster Raster Elevation Scanned Map Time Series Raster Attributes
Scanned Maps Topography Data Model Cartography Administrative areas Parcels Uses, rights/interests, and ownership Legal description Corners and boundaries Survey control Vectors roads, rail, buildings Ortho-rectified Imagery Scanned topographic maps
Scanned Map (Example Implementation) National Geographic Society TOPO! data Scanned USGS Topographic Maps 7.5 degree extents (1 : 24000 series) Southern California Demonstration
Scanned Map Raster Data Model Purpose: Base-mapping Update Schedule: Infrequent updates (** Not part of the decision process anymore in 9.0**)
Scanned Map Raster Data Model Scales of Use: Map Scale 1:24,000 = 7.5 min (4 per pixel @ 500dpi) 1:63,360 = 15 min (10.5 per pixel @ 500dpi) 1:100,000 = 30 x 60 min (16.7 per pixel @ 500dpi) 1:250,000 = 30 x 60 min (41.7 per pixel @ 500dpi) Scanned map: useable at the scale it was made Have an inherent scale of use DO NOT get increasingly better at higher scan rates
Scanned Map Raster Data Model Database Design: Raster Dataset Stored as RGB Inputs had unmatching colormaps Converted Single-band with colormap to multi-band RGB (** Not necessary anymore in 9.0**) Lossy compression JPEG 50 Not used for analysis Bilinear Interpolation pyramid resampling Produced the best looking output for display
Scanned Map Raster Data Model Data Properties: Source Data 1000 s TPQs converted to 1000 s TIFFs 1 : 24000 Topographic Maps (scanned by NGS/Topo!) ~ 3 GB of data Geodatabase Data Raster Dataset ~ 10 GB (grows to this size because of RGB storage) ArcGIS Layer Use at 1:24000
Raster Data Model Case Studies Ortho-rectified Imagery Global Raster Raster Elevation Scanned Map Time Series Raster Attributes
Raster Time Series
Raster Time Series (Example Implementation) Hurricane Data Hurricane Mitch (late Oct early Nov, 1998) Shows time slices of hurricane path USGS data AVHRR NDVI data for continent of Africa Every 10 days averaged over the last 21 years Demonstration
Time Series Raster Data Model Purpose: Viewing of same extent at different times Update Schedule: Always being updated Just add new data to a raster catalog (** Not part of the decision process anymore in 9.0**)
Time Series Raster Data Model Scale of Use: Determined by the data In our cases Regional / Continental / Global extents and applications
Time Series Raster Data Model Database Design: Raster Catalog multiple layers, same extent Lossless compression LZ77 Used for visualization, but could be used for analysis Resampling Cubic Convolution
Time Series Raster Data Model Data Properties: Source Data 30 input TIFFs for Mitch >700 BILs for Africa Geodatabase Data Raster Catalog Needs to be a Raster Catalog Same extent, with multiple captures in time ArcGIS Layer Use 9.0 built-in time-series renderer 8.3 custom code on developer samples Regional / Continental / Global scales
Raster Data Model Case Studies Ortho-rectified Imagery Global Raster Raster Elevation Scanned Map Time Series Raster Attributes
Rasters as Attributes of Feature Classes Geodatabase features can store Raster Datasets One or more raster datasets associated with each feature Raster data is stored with the vector data One central location Demonstration
Overall Decisions / Considerations Before loading, consider these issues Compression Purpose of Raster data Base map v. analysis Highest quality v. Disk space Will you serve/sell the raster data? Input raster format Heterogeneous v. Homogeneous Cell size, theme, type, colormap, etc Pyramid resampling Discrete v. continuous Preserve overlapping data?
Conclusions Raster Data Models are simple on their own part of other data models easy to implement and use Follow these best-practices guidelines Prototype your solution first
Evaluations Please fill them out your comments help ESRI become better at meeting your needs each year THANK-YOU!
End of Presentation Thank-you for attending! Open to Questions