An integrated Framework for Retrieving and Analyzing Geographic Information in Web Pages

Transcription

1 An integrated Framework for Retrieving and Analyzing Geographic Information in Web Pages Hao Lin, Longping Hu, Yingjie Hu, Jianping Wu, Bailang Yu* Key Laboratory of Geographic Information Science, Ministry of Education, East China Normal University, Shanghai , P. R. China * Corresponding Author. blyu@geo.ecnu.edu.cn Abstract Most of the information stored in web pages contains geographic context, such as place names, address, and coordinates. Such geographic information often contains great values and is worth retrieving and analyzing. Traditional search engine is limited in its capability of extracting meaningful geographic messages from mass of unstructured and textual source. Even with the retrieved geographic information, specific systems are still needed to show the spatial distributions of such data. This paper presents an integrated framework which can retrieve and analyze geographic information in web pages. This framework integrates the following core functions: geographic information retrieval; geocoding; spatial analysis and statistical analysis. We also demonstrate the effectiveness of this framework by employing it to retrieve and analyze the geographic information from a particular website. Keywords-web pages; geographic information retrieval; spatial analysis I. INTRODUCTION In recent years, we witnessed a considerable increase in the application of geographic information in disaster prevention and control and regional economic planning. It is vital to keep the relevant database up to date for practical applicability. Since the Internet is a great resource of frequently updated information with geographic components, it is natural to be targeted for data mining. However, the information retrieved from web is usually of textual format. As a cartographical principle, geographic information is better understood in visual format rather than in textual format. Often presented as thematic maps and statistics images, this visual format is commonly used to reveal spatial patterns, inspire spatial associations, and support spatial decision making. Since the geographic information contained in web pages has such great meaning and the visual format is indispensable, it s significant to establish an integrated framework which can retrieve geographic information from the web, convert it into visual format for further analysis and support limited spatial analysis functions. Then, there are three difficult points have to be overcome. First is the accuracy of retrieved geographic information. Second is converting the derived geographic location information to GIS vector data. The last one is the integration of GIS spatial analysis functions. For retrieving geographic information, the concept of geographical information retrieval (GIR) has been proposed as the extension of the field of information retrieval (IR) which featured in the use of an information dictionary [1]. It is concerned with improving the quality of geographically specific information retrieval with a focus on access to unstructured documents such as those found on the Web and a number of researches have been conducted to improve the performance of geographic information retrieval [2]. The location names in web contents are focused in one research because these names often have metonymic meaning different from their literal geographic sense (e.g. Yesterday, Seoul and Peking agreed to start diplomatic relations) [3]. Another research focused on the errors in Natural Language Processing (NLP), and how these errors may influence the performance of GIR [4]. These methods of semantic analysis mentioned can improve the accuracy indeed but they restrict the efficiency of searches when they are practically executing. To avoid complicated semantic analysis and improve the accuracy, the method of professional information collection systems which based on the HTML tags analysis has been introduced into this paper [5]. In addition, a geographic information dictionary is established for supplementary and the target web sites are limited into specific ones such as professional sites with geographic coordinates, yellow pages-like sites and search engine results pages (SERP). Data gathered from the web above is usually unstructured and textual, whereas GIS works only on well-structured and numerically coded data supplied by a spatial database. In order to realize the conversion from text to map, geocoding-the process of associating an address record with a point on a map-is necessary. Gerard Rushton summarized three major methods of geocoding. First method assigns an observation to a geographic unit which is called as address conversion and landmarks conversion in this paper. The second and third methods (interpolation and parcel matching) attach a point coordinate value to a record [6] which is called as latitude and longitude conversion. All these methods are realized aim at the conversion of different types of specific web sites. In addition, the accuracy of geocoding is determined by the input address-based data quality. The paper by Zandbergena discussed the geocoding quality by using different street network Supported by the National Basic Research Program of China (973 Program) (No. 2010CB951603) /11/$ IEEE

2 datasets to geocode the same input files and this method require a researcher to manipulate address-based data in a Geographical Information System [7]. In this paper, the emendation of landmarks is realized for improving the quality of base data. System integration of GIS spatial analysis functions can be conducted in two ways. One is to integrate a modified web data capture algorithm into a commercial GIS, such as ArcGIS (an ESRI desktop product) [8]. Although this approach is convenient and efficient, most GIS functions will not be utilized, leading to a significant resource waste. The other approach is to custom design an independent framework to meet users specific requirements. This way requires the least system functionality and ensures no idle resource during system operation. ESRI s ArcEngine [9] presents an excellent example for this custom system model and was therefore selected as development platform in this study. The objective of this paper is to resolve the aforementioned difficulties and develop a prototype system for geographic data capture from the web and format conversion for GIS database updating, analysis, and visualization. The remainder of this paper is organized as follows. Section 2 details the process of retrieval and conversion methods. Section 3 introduces the prototype system and chooses some specific web sites as case studies. These cases are utilized to demonstrate the methods and implementation. Section 4 concludes this paper. II. METHOD Geographic information retrieved in this paper is mainly coordinates (know as latitude and longitude), address, and landmarks. These data is transformed as shown in Fig. 1. Figure 1. Flowchart of data processing A. Source Code Analysis For accuracy of retrieved address and coordinates, we choose web pages that have a certain rule. These specific web pages include pages from professional geographic site, yellow pageslike site and the search engine results pages (SERPs). In these specific web pages, Geographic information has been stored in semi-structured source code, as is shown in TABLE I. TABLE I. SOURCE CODE MODE OF CHOSEN WEB PAGES Professional site Yellow pages-like site Search engine results tag> tag>event_1/tag> tag>latitude_1/ta g > tag>longitude_1/t ag> /tag> tag> tag> Event _2/tag> tag>latitude_2/ta g > tag>longitude_2/t ag> /tag> tag> tag> Event _N/tag> tag>latitude_n/ta g > tag>longitude_n/ tag> /tag> tag> tag >Name_1/tag > tag >Address_1/tag > /tag> tag> tag >Name_2/tag > tag >Address_2/tag > /tag> tag> tag >Name_N/tag > tag >Address_N/tag > /tag> tag> Keyword_1Key word_2 Keyword_NLan dmark_1 /tag1> tag> Keyword_1Key word_2 Keyword_NLan dmark_2 /tag> tag> Keyword_1keyw ord_2 Keyword_NLan dmark_n /tag> 1) HTML Tags Analysis For pages in professional site and Yellow pages-like site, one can cut out needed address, latitude and longitude from these pages by analyzing the source code and customizing rules based on HTML (Hypertext Markup Language) tags. For example, the user can find out the tag before and behind Address_1. The pair of tags must be unique in case irrelevant information. They also should apply to Address_2 for processing batch. After confirming the tags, N addresses can be cut out from one single page. If there are M pages, [P ((M-1)N) P MN)] addresses can be obtained. Then, the Event_1, Name_1 or other information corresponding to relevant geographic information can be cut out in the same way according user s need. The collection result is shown as TABLE II. 2) Landmark Match As is shown as TABLE I, it s impossible to extract the geographic information by analyzing HTML tags when geospatial information is combined with the other information. So, Landmark match become a feasible method. We cycle the data table of prepared base maps to match the landmarks. The processing result is shown as TABLE II.

3 TABLE II. Retrieving Results of Source Code Professional site Event_1 Latitude_1 Longitude_1 Event _2 Latitude_2 Longitude_2 Event _N Latitude_N Longitude_N Yellow pages-like site Search engine results Name_1 Address_1 Name_2 Address_2 Name_N Address_N Keyword_1 Keyword_2 Keyword_N Keyword_1 Keyword_2 Keyword_N Keyword_1 Keyword_2 Keyword_N Landmark_1 Landmark_2 Landmark_N Now, pieces of meaningful geographic information have been able to be retrieved from these specific web pages. Next step is try to convert these pieces of information into vector map with further handling according the data transfer standards of common GIS. B. GIS Data Conversion The concentration management of Common GIS data is tabular form. The data structure is shown as TABLE III. FID field is a globally unique identifier (GUID) automatically generated at the time of conversion. Shape filed record the shape of the geographical entity (point, line or polygon). It is also known as the spatial location information [10]. Other thematic attribute information of geographic entities is stored in Property_1 to Property_N fields. So, the spatial information and attribute information of the geographic entities must be extracted for converting the web geographic information into GIS spatial data. TABLE III. The Typical Table of GIS Data FID Shape Property_1 Property _N 1 Geometry Geometry N Geometry 1) Latitude and Longitude Conversion Point on Earth s surface is usually described by latitude and longitude. With reference to the appropriate geodetic coordinate system, space orientation of the point can be realized by using its coordinate [11]. The localization principle of GIS is also base on the coordinate. So, latitude and longitude (TABLE IV) can be directly located and converted to a vector map though only a little of professional web site record the latitude and longitude, such as CEDC (China Earthquake Data Center) [12]. 2) Addresses Conversion Compared with the other two types, the conversion of addresses is complex and they cannot be directly converted into vector unless using the method of geocoding. Geocoding, also known as address matching, is the process of creating the corresponding vector map by establish some connection between address table and related base map, associating an address record with a point on a map [13]. First, an address table and a base map are essential data for geocoding. Second, each record in address table must contain explicit road name and house number for address location. Third, attribute information of must be one to one correspondence to the addresses. The format of conventional addresses of China is XX City + XX District + XX Road + XX Number. The road name and the number can be extract by locate the keywords, such as Road, Street, No. or alleyway etc. 3) Landmarks Conversion Landmarks (TABLE IV) are also easy to covert because this kind of geographic information is retrieved by matching exiting base maps. So, landmarks can be converted directly by connecting to relevant base maps. TABLE IV. Professional site Final Tabular form of Retrieving Results Search engine results ID Event Lat Lon ID Message Landmarks 1 E_1 Lat_1 Lon_1 1 K_1K_N L_1 2 E_2 Lat_2 Lon_2 2 K_1K_N L_2 N E_N Lat_N Lon_N N K_1K_N L_N Yellow pages-like site ID Name Address Road Number 1 N_1 A_1 R_1 No_1 2 N_2 A_1 R_2 No_2 N N_N A_1 R_N No_N III. PROTOTYPE AND APPLICATION To meet the requirements above and implement the method mentioned, we designed a prototype system with two main function modes: information retrieve and spatial analysis. The simplified system flowchart is shown as Fig.1. Each function of the system can be divided into several sub-modules. The completed system architecture design is showed as Fig.2.

4 Figure 2. Simplified system flowchart information of SERPs will be match to landmarks. These landmarks are stored in SQL2000 and can be emendated by a menu item in the prototype for additions and deletions. The collection results can be converted by landmarks match base on prepared base maps. The example is shown as follow: The latest information of fire disaster in Shanghai center area. Figure 5. The latest information of fire disaster in Shanghai, China Figure 3. Detailed system architecture design A. Geographic Information Retrieval Module This module can collect geographic information from web pages contents and converting the unstructured geographic data to vector data. There are three ways for user to get geographic information in web pages can. 1) Specified Site Collection One can choose specified sites including China Earthquake Data Center (CEDC) and Dianping.com (a yellow pages-like site) [14] directly by mouse click without any other edit. The HTML tags has been analyzed and edited into build-in collection rules. The final table of results can be converted by coordinate localization and geocoding in accordance with the tabular from. Geocoding is executed by the Address Locator created with the help of ArcGIS Engine 9.3 software development kit [15]. The example is shown as follow: Collection of earthquake information from CEDC. 3) Customize Rules Collection Collection rules for how to cut out the needed information between HTML tags can be customized by users in this submodule. In consideration of the difference between coordinates (latitude and longitude) and Chinese conventional addresses, there is one more step to handle address compared with coordinates. So, two tabular forms can be selected and the final address tabular form will be created automatically by build-in rules. Then, the retrieved results can be converted as same as the specified site collection. The example is shown as follow: The latest vegetarian restaurants in Shanghai center area. Restaurant Address Restaurant Name Figure 6. The latest vegetarian restaurants in Shanghai center area Figure 4. Collection of earthquake information from CEDC 2) SERPs Collection In this module, one can retrieved geographic information from SERPs by input keywords and choose the search engine prefer. There are Baidu and Google as options. Geographic B. Statistic Analysis and Spatial Analysis Module The retrieved geographic information above is converted and stored in geodatabase as point feature class which also known as the point vector layer. According to the spatial characteristics, attribute characteristics and interaction of point features, the analysis functions integrated in this system include regional statistical analysis, network density analysis, buffer analysis, hierarchical thematic map, histograms and attribute query. Thematic map and charts can be created by these sub-modules so that spatial distribution can be reflected and server for decisionmaking. 1) Regional Statistic Analysis

5 According the topological relations between two point feature classes, one can statistically calculate the number of geographical factors in interested regions. The results can be show in the form of thematic map. The example is shown as follow: Regional statistical analysis of vegetarian restaurants in Shanghai center. Figure 9. Magnitude 8.8 earthquake in Chile affected areas (200km) Figure 7. Regional statistical analysis of vegetarian restaurants in Shanghai center 2) Network Density Analysis Network density analysis means calculating the density of geographical factors in a specific grid cell. This function is used to study the spatial distribution of geographical phenomena. The vector grid is overlay analyzed with the target layer according to the size of vector grid and the analysis results can be shown in the form of hierarchical thematic map. The example is shown as follow: Density analysis of vegetarian restaurants in Shanghai center. 4) Hierarchical Thematic Map This sub-module is different from regional statistic analysis as just one feature class is used for statistic analysis. Meanwhile, this function can be used to calculate the maximum, minimum, mean, variance and sum of each numeric filed. The result of hierarchical thematic map is similar to regional statistic analysis. 5) Histogram This sub-module use ZedGraph [17] as chart controls. The results of spatial statistical analysis are reflected in the form of histogram instead of thematic maps. This is another visual expression for users. The example is shown as follow: Histogram of vegetarian restaurants in Shanghai center. Figure 10. Histogram of vegetarian restaurants in Shanghai center Figure 8. Density analysis of vegetarian restaurants in Shanghai center 3) Buffer Analysis Buffer analysis is used for identifying areas surrounding geographic features. The process involves generating a buffer around existing geographic features and then identifying or selecting features based on whether they fall inside or outside the boundary of the buffer [16]. The generated buffer graphics can be outputted as graphics files. The example is shown as follow: Magnitude 8.8 earthquake in Chile affected areas (200km). 6) Attribute Query Attribute query function is similar to the SQL query of relational database. Users can query interested geographic information in attribute tables of feature classes. The results will be highlighted on the map. IV. CONCLUSION This paper has proposed an integrated framework designed for retrieving and analyzing geographic information in web pages. It can retrieve web geographic information and convert the derived geographic location information to GIS vector data by using ArcGIS Engine 9.3 software development kit. Meanwhile, the vector data are organized and stored in a geodatabase and displaying on the map control for further spatial analysis and

6 statistical analysis. Finally, tailor-made graphics such as thematic map and statistics images are outputted by the prototype for practical decision-making of users. However, this system is just a prototype for demonstrating the feasibility of above function. The base map we choose is limited for large area application and the prototype is target mainly to Chinese specific web site. Further study must be made for retrieving more types of web site, improving search efficiency and expanding the analysis modules. REFERENCES [1] X. Wang, Research on Geographic Information Retrieval, University of Science and Technology of China, [2] C. B. JONES and R. S. PURVES, Geographical information retrieval, International Journal of Geographical Information Science, vol. 22(3), 2007, pp [3] L. Johannes and H. Sven, On metonymy recognition for geographic information retrieval, International Journal of Geographical Information Science, vol. 22(3), 2008, pp [4] S. Nicola, L. Yi, M. Alistair, and R. Jiawen, An empirical study of the effects of NLP components on Geographic IR performance, International Journal of Geographical Information Science, vol. 22(3), 2008, pp [5] K. Zhou, Exploring the Integration of GIS and IR and its Application for the Management of Emergency Response, East China Normal University, [6] G. Rushton, Geocoding in Cancer Research, American Journal of Preventive Medicine, 2006, vol. 30(2S). [7] P. A. Zandbergena, Influence of street reference data on geocoding quality, Geocarto International, 2011, vol. 26(1). [8] Esri Products, accessed at April 20, [9] ArcGIS Engine Overview, accessed at April 20, [10] C. Zhang, Internship Guide of GIS, Higher Education Press, 2000, pp [11] M. Cai, New Cartography Tutorials, Higher Education Press, 2006, pp [12] CEDC, accessed at April 20, [13] P. A. Zandbergena, Influence of street reference data on geocoding quality, Geocarto International, 2011, vol. 26(1). [14] Dianping.com, accessed at April 20, [15] Engine Help for.net (VS2008) [16] Buffer Analysis, accessed at April 20, [17] ZedGraph, accessed at April 20, 2011.