Prosurv LLC Presents

Size: px

Start display at page:

Download "Prosurv LLC Presents"

Barnaby Blair
5 years ago
Views:

Data usually consists of dozens, hundreds, or even thousands of points.

1 Prosurv LLC Presents An Enterprise-Level Geo-Spatial Data Visualizer Part IV Upload Data Upload Data Click the Upload Data menu item to access the uploading data page. Step #1: Select a Project All Projects need data. Data usually consists of dozens, hundreds, or even thousands of points. Each point represents a location, and thus, it has a set of coordinates, such as latitude and longitude. If you ve read some of the other guides in this series, you know that latitude and longitude are known as spherical coordinates, as opposed to rectangular coordinates. Page 1

2 Coordinate Systems A Primer Almost every coordinate system is 3-dimensional meaning that it has an X, Y, and Z component to it. In a spherical coordinate system, like the one designed to fit our Earth, Longitude is the X coordinate (East-West), and Latitude is the Y coordinate (North-South). The Z coordinate is the Elevation, which is sometimes referred to as the Geoid Height. In Land Surveying, rectangular coordinate systems are often used, rather than spherical ones. It is much easier to work with rectangular coordinate systems, than it is to work with spherical ones. For one, calculations are much easier. Surveyors use what is known as Coordinate Geometry to solve equations and come up with solutions. So, in Surveying, when working with rectangular coordinates, Northings are the Y values, and Eastings are the X value. A point is usually described as: Point #, Northing, Easting, Elevation, Description For example, you might have a text file where each line represents one point, like: 1, , , , Bldg X This is known as a comma-delimited file, a.k.a. a CSV file (used by Microsoft Excel, CSV stands for comma-separated-values). Surveyors sometimes refer to these types of text files as points files, or ASC files, since they used to be given the extension.asc. Notice that each point is Northing first, followed by the Easting. This means that the points were actually Y,X instead of X,Y. Funny, I know, but that s how surveyors did things. And Y meant North, so the first value was how far North of the other values, where X (Easting) was how far East of other values. In a local coordinate system, a surveyor might simply be surveying your house, and thus has no need to use something as fancy as a State Plane Coordinate system. Instead, they might just use a local or small coordinate system. For example, they would just start by setting a point in the ground and calling it 5000 N, 5000 E. Satellite imagery is a whole different ballgame altogether. First, you re trying to show the entire globe projected onto a flat surface that s what all paper maps do they project spherical data onto a 2d object. It all comes down to how you compare two different points, or locations, to each other, on a map. Page 2

3 In the United States, a system was developed, called the State Plane Coordinate System, or SPC for short, so that state by state, data could be accurately plotted and compared to other data in that state. Surveyors like to have coordinates that always stay positive. Go too far West, however, and the X coordinate decrease to a point where they become negative. Go too far South and you get the same thing with the Y coordinate. So, State Plane systems have an origin or starting point that is BIG. That way, all values always remain positive. Another rectangular coordinate system is the Universal Transverse Mercator system, a.k.a. UTM for short. The UTM system consists of 60 zones and covers the entire World. Each zone is 6 degrees in width in Longitude (6 * 60 = 360 degrees). There is of course UTM North, for the northern hemisphere, and UTM South, for the Southern hemisphere. Each UTM zone also has an origin point with large starting values for Northings and Eastings (Y and X). Again, this is to keep the values positive, anywhere within that zone. Now, technically, you don t need to care about any of this stuff. But it s nice to know some of these things, if you don t already. Hopefully, all you need to do is get a hold of some data, usually as some sort of file, and then simply upload the file(s) to your Project, process the file(s) and you re done! If you know that your project is simply Latitude and Longitude, then all you ll do is upload the files. In your project definition page, you ll tell Prosurv GeoSpace DataViz that your project is Latitude and Longitude. Then, just upload your files. With GPX files, it s even easier. GPX files and the data within them, are always in latitude and longitude, and their elevations are always in meters. So talk about easy you just upload them and you re done. Page 3

4 File Formats However, if you ll be uploading text files or csv files (a CSV file is really just a text or notepad file that is comma-delimited, i.e. separated by commas hence the name Comma Separated Values), then you need to know the proper format that Prosurv GeoSpace DataViz is expecting. Text or CSV files that are uploaded, must be in the following format: Point #, Northing, Easting, Elevation, Description The Description field is sometimes known as or referred to as the Feature Code field. Any uploaded file must have at least these 5 fields, separated by commas. You can have up to 10 fields in total, with the last 5 fields being Attributes fields. Attributes contain extended information about a point. Each attribute can be up to 255 characters in length. You can actually build a coordinate file on your own. Just open up Notepad on your PC, and enter one line for each point. Start by giving the point a point number. Then hit the comma key, then enter the northing for that point (again, either a State Plane northing, or, a UTM northing). Then hit another comma, then enter the Easting. Be sure to put an elevation, even if it is just zero (0). And finally, give the point some type of description. Avoid the use of a comma in the description, or else the app will think it s another attribute, instead. Uploading GPX Files GPX files are really easy. With a GPX file, you don t have to worry about any selections prior to upload. Just browse for your file, upload the file, and then select and process the file. That s it. Again, that s because GPX files are always defined using latitude and longitude, and they are always in metric units. Uploading.txt or.csv Files Before uploading a text (*.txt) or CSV (*.csv) file or an old-fashioned.asc surveying file, you must be sure that you ve made the right selections that tell the app what specific coordinate definition is being used. In the Manage Projects screen, you select whether the project data will be latitude/ Page 4

5 longitude (Yes or No). By setting it to No, then will presume that all of your coordinate files will be rectangular coordinates. So, step #1 is telling the Project whether it is Latitude/Longitude or not. If your project is NOT defined by latitude/longitude, then you need to tell the app whether the rectangular coordinates are UTM or State Plane. If they are State Plane, then you will select the State and the Zone. If they are UTM, then you ll simply select the UTM Zone, such as UTM 13. Now, your Project will know the coordinate system that the data inside your files, is using. Text / CSV Files Units In the Manage Projects screen, as we ve just described, you set the coordinate system (spherical vs. rectangular and UTM versus SPC) being used. And that is applied over the entire Project. However, individual files that you upload in the Upload Data screen, might have different units. For example, you might have a file that was given to you in metric units, and another file that was given to you in US foot coordinates. For those wondering about US Foot coordinates, the conversion factor between meters and US Foot is exactly 1200/3937. So 1 meter = 3937/1200 US feet. This is different than the conversion between the international foot and meters, where 1 foot =.3048 meters exactly. When working with big, large, state plane and UTM coordinates, the difference does matter. All surveying that occurred within the United States was based on the US foot, not the international foot, which was defined later, in By the way, 3937/1200 = , or inversed, 1200/3937 = , so note the difference between.3048 exactly versus the US foot definition. But again, when multiplying by a coordinate value that s in the hundreds of thousands, or millions, it DOES make a difference. Too much information, right?! Again, you really don t need to know this, but Page 5

6 Here s another interesting fact: Ever wonder why mile markers on the Interstates go up to the East and down to the West (and up to the North and down to the South)? Well, that follows the same logic as surveyors and their Northings and Eastings coordinate values. Eastings always increase when going East, i.e. the coordinate value gets bigger as you go East. Northings always get bigger going North. And they get smaller (decrease) going South. So when you drive East in Nebraska along I-80, the mile marker signs will increase, up to a larger value. They go from 1 to around 455. But heading West, the mile markers go down and eventually go to zero. What to do if you have coordinates, but don t know what they are? In some industries, you may be given a set of engineering plans for your Project. These engineering plans should contain the Coordinate Definition, in other words, the words describing what projection system and the zone the coordinate values are in. For example, the set of plans might say Wyoming West Central. Okay, great. Now you know what to select in the Manage Projects screen. But what if you don t know? What if you re given some data files, and they ve got these large numbers that are clearly NOT latitude and longitude, but you re not told what zone they re in, or even what coordinate projection system (i.e. UTM vs. SPC)? Well, some quick research might help you out. Obviously, you should know what city, state, and most importantly, what county the project is in. Then, you could do a web search for, say, Nolan County, Texas. Type in Nolan County TX State Plane Zone and see what comes up. Aha! Here we go: Page 6

So, now you can clearly see what counties in TX are in what SPC zone! SPC zones are almost always defined by county lines. And you can see that this is the case in TX.

So, you would select TX North Central from the dropdown lists in the Manage Projects screen. Then, you would upload the data file(s).

7 So, now you can clearly see what counties in TX are in what SPC zone! SPC zones are almost always defined by county lines. And you can see that this is the case in TX. So if your project is in Nolan County, TX, then you can be assured that the values that you ve been given, are most likely TX North Central. So, you would select TX North Central from the dropdown lists in the Manage Projects screen. Then, you would upload the data file(s). If everything looks correct, once the data is plotted on the satellite imagery, then you can assume that you ve chosen the right coordinate system. If not, then it s very possible that your data is in UTM coordinates, instead. Page 7

8 Once again, a quick web search for Texas UTM Zones yields: So now you know that Nolan County, TX is in UTM Zone 14. Pretty easy, right? So you d go back to your Manage Projects screen, and you d select UTM 14. Then hit the Update button. If the data still doesn t look right when it s plotted, then it may be a Units issue. Perhaps the units are in feet, rather than meters. How can you tell? Well, if the data has elevations, which it probably will, then you might be able to go by the size of the elevation. Perhaps look up the nearby town and do a wiki search for the town information. There should be an elevation listed for that town or a nearby airport. Of course, depending on what state you re in, elevations may change quickly in just Page 8

a few miles, so a meters vs. feet comparison of elevations to try to determine whether your coordinates are in feet or meters, might not always work. But it s worth a shot.

9 a few miles, so a meters vs. feet comparison of elevations to try to determine whether your coordinates are in feet or meters, might not always work. But it s worth a shot. If all else fails, once you ve got your Zone pinpointed, and it s just a matter of Units, you could always upload the data and see where it plots. 99% of the time, if the Units are not correct, then the plotting will be WAY off. Now, I ve tried to go into detail here, in the event that you re stuck and the data doesn t seem to work. But I don t mean to over-complicate things. In fact, creating a Project and uploading and processing data for that Project should take you less than two minutes, tops. And if you re using GPX files, then the whole thing should take you less than 30 seconds! Finally Uploading Some Data! Page 9

10 Uploading data for your Project is a simple, 3-step process: 1. Browse for your file. 2. Upload the file to the cloud. 3. Select the file(s) that you want to process, and hit the Process button. So you select and upload one file at a time. But then, you can process any number of files, all at the same time, by checking the box for each file and then hitting the Process Selected Files button. uses a unique system to store your files in the cloud. First, it creates a folder, unique to your Company. Next, it creates folders inside your Company folder, based on a unique Project Id. This way, all of the files that you ve uploaded for a Project are maintained and stored inside a folder just for that Project. So, when you select a Project, you instantly see all the files that have already been uploaded for that Project. Also, let s say you forgot to set the Units correctly before processing a file. Well, fixing that is easy. Whenever a file is processed, looks for any data that was associated with that file, and deletes it automatically. This way, your points won t be duplicated, just because you ve processed the same file, twice. Finally, before processing any files, be sure to select from the two dropdowns below. In other words, be sure that your Units are set correctly, and be certain that you ve selected whether the Latitude and Longitude (if you re uploading Lat/Long files) are properly defined as either being DMS (Degrees-Minutes-Seconds) or Decimal Degrees. So let s go over the entire process, start to finish Page 10

11 Step 1: Select your Project Start by selecting the Project that you re uploading data for. Prosurv GeoSpace DataViz is designed to store your data files inside a project folder (which is inside a company folder) in the cloud. They stay there forever, until you delete them. This means that you can always go back and reprocess any file that you ve uploaded before. Notice that when you select your project, the spreadsheet is updated with your existing list of files that have been uploaded, previously. An interesting and bonus capability, is that you can Download any file that s been uploaded previously. Simply click the Download link in the appropriate row. In a way, the bonus is that you can share the data file with any other authenticated user (of course, it has to be someone with the appropriate Role/permissions). Think of it as a way to a data file to a colleague, without actually having to it! Note that the Project Coordinates Definition is shown. In this case, it is telling you that for this project, you ve set it to Latitude, Longitude in the Manage Projects screen. Page 11

Step #2: Select the File to Upload To upload a new data file, click the Choose File button. Now, browse for the file on your computer, then hit the Open button (or, just double-click the filename).

12 Step #2: Select the File to Upload To upload a new data file, click the Choose File button. Now, browse for the file on your computer, then hit the Open button (or, just double-click the filename). Note that the button now reflects your selection. Step #3: Upload the File to the Cloud Click the Upload File button to upload your file to the Cloud. The list will now repopulate, showing you that the file has been uploaded. Page 12

13 Step #4: Process the Selected Files* Click the Process Selected Files button to process one or more files, all at the same time. You can select multiple files by clicking the checkbox in the appropriate row. You can select all of the files by checking the box in the header row (top row). Then, you can de-select all the files by unchecking that same box. It will only take a second or two for each file to be processed. After processing your file(s), you can click the Back menu item to return to the Overview screen. If this is the first set of data to be uploaded for a project, then you should now see a single point representing that project, in the Overview. Now, click on that point and voila! You re now looking at all the data in your project! *NOTE: Before processing your file(s), be sure to follow this chart: If your data files are GPX files, then there s nothing to worry about...just process them and you re good to go. If your data files are either Text (*.txt) or CSV (*.csv) files, then; If your file contains spherical coordinates, that is, latitude and longitude, be sure that you ve selected whether the values are DMS or Decimal degrees. Also, see the section on the proper format for the data file. If your file contains rectangular coordinates, that is, either State Plane or UTM coordinates, be sure that you ve selected the proper Units, i.e. Meters, US Foot, or International Foot. Page 13

When uploading a file containing rectangular coordinates, be sure to select the Units, prior to processing the file(s). The Default Units are Meters.

If your Project Coordinates Definition shows Latitude, Longitude, then be sure to select whether the latitudes and longitudes are in DMS or Decimal Degrees, prior to processing the file(s).

14 When uploading a file containing rectangular coordinates, be sure to select the Units, prior to processing the file(s). The Default Units are Meters. So if you don t select it, and the units are actually in meters, then you re okay. If your Project Coordinates Definition shows Latitude, Longitude, then be sure to select whether the latitudes and longitudes are in DMS or Decimal Degrees, prior to processing the file(s). What is DMS versus Decimal Degrees? Well, Decimal Degrees are easy. A Decimal Latitude is just, say degrees. DMS is a bit trickier, because you sort of have to know that the values are in DMS. DMS stands for Degrees-Minutes-Seconds. sometimes shown in this notation: DMS Latitudes and Longitudes are However, this notation doesn t work well inside a text file. So DMS is instead represented properly by entering the degrees (42) followed by a period, and then simply followed by the minutes and seconds values. So the above Latitude looks like: So if your data files are in DMS, then be sure that the latitudes and longitudes are Page 14

15 in the D.MMSSsssss format, as shown immediately above. D.MMSSsssss means Degrees followed by a period, followed by a two-digit value for the minutes, and then a two digit value for the seconds, followed by up to 5 digits that represent the decimal seconds. In other words, the seconds are but you leave out the period separating the integer seconds from the decimal seconds. Note that the above values are the exact conversion between DMS and Decimal. Use this formula to check: Decimal Degrees = Degrees + (Minutes/60) + (Seconds/3600) is Powerful Taken altogether, the capabilities of are pretty powerful, in terms of being able to upload virtually any type of coordinate data, and having the app process this data correctly and very accurately. Everything that you see in the Overview and the Project View is data that is displayed based on spherical coordinates (latitude/longitude). So what needs to happen is that if you re uploading rectangular coordinates (SPC or UTM), Prosurv GeoSpace DataViz actually converts those values into Latitudes and Longitudes. And if your data is Lat/Long in DMS, it converts those values to Decimal Degrees. If your data is in some type of foot units, then even that is converted as well. In fact, your data might be US Foot State Plane, and yet, has the power and capability to convert the values into true and very accurate latitude and longitude values. In fact, here s a listing of the data that can process: 1. Decimal Latitude and Longitude 2. DMS Latitude and Longitude 3. UTM Coordinates in Meters, US Foot, or International Foot 4. State Plane Coordinates in Meters, US Foot, or International Foot 5. GPX Files (no definition necessary, since GPX files are always in Decimal Degrees Latitude and Longitude and always have metric elevations). Page 15

16 Proper Data File Configurations Text and CSV files (.txt/.csv) must be in one of the following formats (Feature Code, also known as the description of the point, is shortened here to FC ): Point #, Northing, Easting, Elevation, Feature Code* Point #, Northing, Easting, Elevation, FC, Attribute 1 Point #, Northing, Easting, Elevation, FC, Attribute 1, Attribute 2 Point #, Northing, Easting, Elevation, FC, Attribute 1, Attribute 2, Attribute 3, Point #, Northing, Easting, Elevation, FC, Attr 1, Attr 2, Attr 3, Attr 4 Point #, Northing, Easting, Elevation, FC, Attr 1, Attr 2, Attr 3, Attr 4, Attribute 5 What Does Processing Mean? contains some very powerful AI-style routines. When you process one or more files, the app actually scans each file for common feature codes (descriptors), and then matches them up with other points that have the same codes. GPX files can contain treks in other words, like a path connected by dots. So again, processing GPX files are pretty easy, because they already define treks as well as just separate points, known as Waypoints. But text files don t make these connections, themselves. So, the AI built into Prosurv GeoSpace DataViz, compares feature codes and creates Sets of points with common codes. The points are assumed to be in sequential order, however, they don t have to all be bunched together within the file. In other words, points 1 through 5 might have Edge of Pavement 1 or EP 1 for short. Then, points 6 through 14 represent a building, with a feature code of Bldg 1. Then, points 15 through 35 might be more EP 1 points. Well, Prosurv GeoSpace DataViz, automatically scans the file and builds these Sets of points thus creating lines, much like the treks in a GPX file. And it does all this, automatically! Page 16

17 A single file might generate 25 or 100 or even more Sets of points, which are then drawn all as separate lines. Each of these lines can then be altered in terms of their line color and width, by assigning a line definition to each separate line! But the nice thing is that it is all automatically done for you. See Part V Plotting Options to learn more about setting up your Line Definitions, Colors of Lines, Width of Lines, Colors of Points and Point Types. Page 17

Moving into the information age: From records to Google Earth

Moving into the information age: From records to Google Earth David R. R. Smith Psychology, School of Life Sciences, University of Hull e-mail: davidsmith.butterflies@gmail.com Introduction Many of us