Dan Goldberg GIS Research Laboratory Department of Computer Science University of Southern California

Transcription

1 Presented to: CSCI 548 Information Integration on the Web Dan Goldberg GIS Research Laboratory Department of Computer Science University of Southern California 1

2 (Very) Brief Background Locational descriptions Geographic representations USC GIS Research Laboratory 3620 South Vermont Ave, Los Angeles, CA Kaprielian Hall, Room 444 Los Angeles, CA Spatio-Temporal Analyses 2

3 Motivations 3620 S. Vermont Ave, Los Angles CA GEOCODER (Google, ESRI, Yahoo, Geocoder.us, etc.) $$$ $$ $ Parcel Centroid : , Address-Level : , Zip Centroid : , $ Time, Money, Expertise, Programming 3

4 Motivations Error introduction/propagation in epidemiological research Relative Magnitude Error Propagation Address Locational Spatially Referenced Geocode Address Calculate Exposure Incomplete / incorrect Inaccurate location Incorrect assignment Values Spatial Analysis Invalid association Hot Spots Conclusions Misguided actions

5 Motivations Exposure misclassification from inaccurate geocoding Misclassified exposed Misclassified unexposed distribution area zip code 1 zip code 2 address range geocode zip centroid geocode point source 5

6 Motivations Accessibility mischaracterization from inaccurate geocoding zip code 1 zip code 2 address range geocode zip centroid geocode true shortest path false shortest path The error from geocoding can be larger than the distance traveled 6

7 Motivations Geocode quality the type of reference feature matched 3620 S. Vermont Ave, Los Angeles CA = NAACCR GIS Coordinate Quality Codes Code Value 1 Global Positioning System 2 Parcel centroid 3 Complete street address 4 Street intersection 5 Street segment mid-point 6 ZIP+4 7 ZIP+2 8 Manual Assignment 9 ZIP 10 PO Box / Rural Route ZIP 11 City Centroid 12 County Centroid 98 Assigned, quality unknown 99 Non-assignable a) Building b) Parcel c) Street segment d) Relative direction e) ZIP+4 f) City The geocoding process needs to be taken into account g) County h) State 7

8 Motivations All geocodes with same quality do not have the same accuracy or certainty 2562 Elmwood St, Los Angeles CA Attribute Relaxation (Suffix) Soundex(Name E453) Parcel centroid 2562 Ellendale Pl, Los Angeles CA Parcel centroid 2562 N Ellena St, Los Angeles CA Attribute Relaxation (Pre, Suffix) Substring (Name - Ellen) Parcel centroid Qualities of the feature matching method matters Correct Incorrect 8

9 Motivations All geocodes with same quality do not have the same accuracy or certainty NAACCR 2: Parcel Centroid Bound Box: Geometric: Weighted: Address range: X NAACCR 3: Street Address Uniform lot: Actual lot: Y Y*d X *d X Qualities of the feature interpolation matters 9

10 Motivations All geocodes with same quality do not have the same accuracy or certainty :12000 scale :63360 scale : scale Qualities of the reference features matter 10

11 Motivations 3620 S. Vermont Ave, Los Angles CA GEOCODE , Accuracy =?? Match rate of geocoder used =?? Spatial uncertainty of this geocode =?? Reference data used to produce this geocode =?? Interpolation assumptions used to produce this geocode =?? Average spatial uncertainty for other geocodes in the area =?? 11

12 Theoretical and Technical Contributions 1) A theoretical and practical framework for creating and evaluating geocode components Enables 2) A method for evaluating overall geocode accuracy as a composite function of its internal processes Measures 3) Several novel methods for improving geocode accuracy Predicts and Improves 4) A method for predicting and improving the accuracy of geocodes based on local and neighborhood characteristics 12

13 Theoretical Framework Input Normalization/ Standardization Matching Reference 3620 South Vermont Avenue 3620 S VERMONT AVE Transform input to match reference data format Find a matching geographic feature in reference data SELECT FromX, FromY, ToX, ToY FROM SOURCE WHERE (Start >=3620 AND End <= 3620) AND (Pre = S) AND (Name = VERMONT) AND (Suffix = AVE) Interpolation Output Use matched geographic feature to derive output Output Point = (20% * X, 20% * Y) 13

14 Component: Input Input Normalization/ Standardization Matching Reference Error Contribution Many different types, forms, and formats: Street Addresses: 3620 South Vermont Ave Postal Codes: Los Angeles, CA Named Places: USC Kaprielian Hall Intersections: Vermont & 36 th Place Relative Descriptions: b/w Bakersfield & Shafter Different levels of information/certainty: Street Addresses: Somewhere on street Postal Codes: Somewhere on postal route Named Places: Absolute location Intersections: Somewhere near intersection Relative Descriptions: Somewhere near locations Interpolation Output Incompleteness: Inaccuracy: 3260 S Vermont 3620 _ Vermont Ave Vermont Ave 3620 S Verment Ave 362_ S Vermont 3260 _ Vermont St 14

15 Component: Input Cleaning Input Normalization/ Standardization Matching Reference Interpolation Error Contribution - Parsing Separating components of the address Token-Based: relies on formatting - Normalization Identifying components of the address Substitution-Based: relies on the token ordering Context-Based: relies on position and schema knowledge Probability-Based: relies on likelihood of occurrence - Standardization Formatting components of the address Schema mapping: must exist for all reference sources 3620 South Vermont Ave Los Angeles, Street Address City Zip St Los Angeles St Los Angeles, Output Street Address City Zip 23 E South St South Los Angeles, Street Address City Zip 15

16 Component: Matching Input Normalization/ Standardization Matching Reference Error Contribution - Multiple Match Types Feature selected from reference set Exact: A single perfect match Non-exact: A single non-perfect match Exact ambiguous: Multiple perfect matches Non-exact ambiguous: Multiple non-perfect matches None: No matches - Multiple Matching Methods Ways of selecting features Deterministic: Rule-based, iterative Probabilistic: Likelihood-based, attribute weighting Interpolation Output - Multiple Fuzzifying Techniques Alter input data Word Stemming: Porter Stemmer Phonetic : Soundex Attribute Relaxation: Remove attributes and retry match 16

17 Component: Reference Input Normalization/ Standardization Matching Error Contribution - Multiple Types Point-based: ZCTA and Place Centroids Linear-Based: Street Centerlines Areal Unit-Based: Parcels, ZCTA and Place Boundaries - Wide spectrum of accuracies/completeness Commercial vs. Public - Attribute accuracy spatial and non-spatial - Attribute completeness spatial and non-spatial - Feature complexity simple vs. polylines Local Scale vs. National Scale - Census Place Boundaries vs. Local Neighborhoods - Wide spectrum of cost/availability Reference Free vs. Costly: TIGER/Lines vs. TeleAtlas Available vs. Not: Address points CA. vs. N. Carolina Interpolation Output Low resolution reference street High resolution reference street 17

18 Component: Interpolation Input Normalization/ Standardization Matching Error Contribution - Many methods of interpolation Depend on reference feature type Depend on info available (assumptions) - Linear-Based: Linear, Segmented, and Summation Address range: X Reference Y X *d Y*d X Interpolation Uniform lot: - number of lots known Output Actual lot: - sizes and number of lots known 18

19 Component: Interpolation Input Normalization/ Standardization Matching Reference Interpolation Error Contribution - Many methods of interpolation Depend on reference feature type Depend on info available (assumptions) - Areal Unit-Based Centroids: Bound Box: Geometric: Output Weighted: 19

20 Component: Interpolation Input Normalization/ Standardization Error Contribution - Lack of Process Transparency - Nothing reported about the decisions made or alternatives Matching Reference - Output Type: Only Geographic Coordinates - Loss data required for determining true accuracy Interpolation Output - Output Accuracy: Feature Match Type + Probability - Nothing that indicates direction - Nothing that indicates distance - Nothing that indicates certainty area or surface 20

21 New Abilities: Quantitative Error Calculation Input and Representation Normalization/ Standardization Probability and spatial uncertainty surfaces for each step Matching Reference Interpolation Output Geocode location with a spatial probability distribution 21

22 New Abilities: 3620 Vermont, Los Angeles Composite Feature Matching with Dynamically Created Features Search Dynamic Feature Composition Ambiguous Results Feature Interpolation N Vermont Ave N Vermont Pl S Vermont Ave S Vermont Pl 22

23 Current Status Production system > 1,000 users 3 million geocodes produced.net implementation on top of SQL server for reference data TIGER/Lines, LA County Parcel Actively adding more parcel data Code is being reviewed, cleaned, and finalized before open sourcing 23

24 Conclusions Geocoding systems need to be open boxes Users need to know what happened, why, and what the alternatives were The commercial alternatives may be too expensive black boxes Existing open source alternatives are Non-extensible The USC WebGIS Geocoding framework aims to achieve these goals by providing an open source extensible approach to approaching the problem Allowing users to 24

25 Thanks! Advisors John Wilson, USC Geography Craig Knoblock, USC Computer Science Myles Cockburn, USC Preventive Medicine 25