Python Tip: Coordinate Clean Up

If you create point features from coordinates stored in flat tables you probably find cases where the longitude and latitude are reversed or a negative sign is missing.  This happens frequently if the source of the data is from a non-GIS user.  What can make fixing these mistakes more difficult is if the errors are not consistent through out the table.  The following Python example will switch the longitude and latitude if necessary and add negative signs if missing.  Note: This code is only useful if all the points are with in the same hemisphere.  If the points are located across the globe, a solution to this problem is much more difficult.

First, import the modules and create the geoprocessor:

import sys, string, os, arcgisscripting

gp = arcgisscripting.create()

gp.OverWriteOutput = 1

Next, create an Update Cursor:

rows = gp.UpdateCursor(r"C:\Project\sample.gdb\table")
row = rows.Next()

Now you need to loop through each row in the table and get the longitude and latitude values.  Note that the XCOORDINATE and YCOORDINATE are the name of the field ins the table:

while row:

     xCoord = row.XCOORDINATE

     yCoord = row.YCOORDINATE

We will need to switch the longitude and latitude if they are reversed.  We do this by checking to see if the longitude is within a valid range.  In this all points should fall with 30 to 50 degrees West Longitude.  If they do not, we assume that they are switched and switch them:

      if xCoord > 30 and xCoord < 50:

          row.XCOORDINATE = yCoord

          row.YCOORDINATE = xCoord
          xCoord = yCoord

We use simpler logic to check for a missing negative sign.  In this case the longitude should be negative so we check.  If it is not, we make it negative:

     if xCoord > 50:

          xCoord = xCoord - (xCoord * 2)
          row.XCOORDINATE = xCoord

Now that the changes have been made if necessary we update the row and move to the next row:

     rows.UpdateRow(row)
     row = rows.Next()

Finally, delete the cursor:

del row
del rows

As you can see the values will change depending on the location of your points, but this general logic should solve most of your problems.  And like all examples, there are several ways to do most of these steps.

Migrating Data from Trimble GPS Pathfinder Office to an ESRI Geodatabase

If you use Trimble GPS Pathfinder Office, you have probably discovered that it does not export data directly into an ESRI geodatabase.  This can be a significant problem, especially if you have a large data dictionary with many features.  However, GPS Pathfinder Office does allow you to export each feature to an individual shapefile.  Using Python, we can quickly move all of the data in the shapefiles into a geodatabase.  The following script will do this automatically for you.  Some important notes, however, are:

1.) This script was written quickly to get the job done.  There are many ways this can be done, and this is only one of them.
2.) The data model of the geodatabase must match the data dictionary exactly.  This is important because the script only works if it does.  Actually, the fields do not have to have the same name, but they need to be in the same order.  You will see below why this is important.
3.) The feature classes in the geodatabase must mach the names of the features in the data dictionary.  For example, if it is called "Roads" in the data dictionary, the feature class shoudl be called "Roads" as well.
4.) All of the feature classes must be in the same data set.

Now let's get started.  First import your modules, create a geoprocessing object, and load the Data Management Toolbox:

import sys, string, os, arcgisscripting

gp = arcgisscripting.create(9.3)

gp.AddToolbox("C:/Program Files/ArcGIS/ArcToolbox/Toolboxes/Data Management Tools.tbx")

Next get a list of all the shapefiles in the output directory from GPS Pathfinder Office:

fileList = os.listdir("C:\\Project\\Data_Dictionary\\SHP")

Now we need to iterate through the list of files in the directory.  We first need to determine if the file is a shapefile by getting its extension:

for i in fileList:

    splitText = os.path.splitext(i)

Now we need to check if the file extension is a shapefile, create and empty string for creating the Append parameter, and start a counter that will be used to keep track of the position of the fields:

    if splitText[1] == fileExt:

        fieldMap = ""

        count = -1

Now we need to get a list of the fields in the matching feature class.  If the shapefile does not match a feature class, it will be skipped:

        try:

            fields = gp.ListFields("C:\\test.mdb\\Data\\" +\

                splitText[0])

        except:

            continue

Let's loop through the fields in the feature class and match them up to the shapefile.  We also need to increase the counter:

        for field in fields:

            count += 1

It is actually not possible to make the data model exactly the same because the feature class requires an OBJECTID and Shape field.  To overcome this we will skip these fields in the field mapping:

            if field.Name == "Shape" or 
            field.Name == "OBJECTID":
                break

We must now get the information from the shapefile:

            else:

                gp.MakeFeatureLayer("C:\\SHP\\" + i,

                    "tempLayer")

                desc = gp.Describe("tempLayer")

                fieldInfo = desc.FieldInfo
               
In the parameter for the Append tool, we need to sepearte each field with a semicolon except for the last field:

                if count > 2:
                    fieldmap += ";"

Now we build our field mapping string from all the information we just collected.  Here you will see we use the counter as the index number of the field:

                fieldMap += field.Name + " '" + field.Name +\
                   "' true true false " +\
                   str(field.Length) + " " + field.Type +\
                   " 0 0 ,First,#,C:\\SHP\\" + i + "," +\
                   fieldInfo.GetFieldName(count) + ",-1,-1"

We can now finish the script by using the Append tool to append the records from the shapefile to the feature class:

        gp.Append_management("C:\\SHP\\" + i,
           "C:\\test.mdb\\Data\\" + splitText[0], "NO_TEST",
           fieldMap, "")

As you can tell, this not necessarily the most efficient way to do this.  I have since rewritten this process using C# to create a more stable tool, however, this fairly simple script can save you quite a bit of time.  I encourage you to modify it and make it work even better.

Python Tip: Iterating through a feature class

After you have mastered creating Python scripts to perform geoprocessing tasks on single feature classes, you will probably realize that you will often come across cases where you will want to perform the same tasks on all the features in a feature class, such as adding a field, adding a prefix or suffix to the name, etc.  Accomplishing this is very easy.  You ill need to iterate through a feature class using the ListFeatureClasses function.  I will give you an example that capitalizes the names of the feature classes in a feature class.  Start by importing the proper modules:

import sys, string, os, os.path, arcgisscripting

Next add the toolbox with the Rename tool:

gp.AddToolbox("C:/Program Files/ArcGIS/ArcToolbox/Toolboxes/Data Management Tools.tbx")

Set the workspace to the feature class you wish to edit:

gp.workspace = "Database Connections\\GIS.sde\\Data"

Call the ListFeatureClasses function and move to the first feature class:

fcList = gp.ListFeatureClasses ("*", "all")
fc = fcList.Next()

Now loop through and rename each feature class:

while fc <> None:

     gp.Rename_management(gp.workspace + "\\" + fc, 

                          gp.workspace + "\\" + fc.upper() \

                          + "_temp", "FeatureClass")
     gp.Rename_management(gp.workspace + "\\" + fc.upper() \

                          + "_temp",  

                          gp.workspace + "\\" + fc.upper(),

                          "FeatureClass") 

     fc = fcList.Next()

Because ArcGIS is not case-sensitive, we had to temporarily rename the feature classes with "_temp" at the end, and then rename them again and capitalize them.  Also, remember to call the Next function to move to the next record or you will be stuck in an infinite loop.

In this case, we ran through every feature class but using the wild card parameter, you can select a subset of the feature classes.  Other similar functions exist that are also useful such as ListDatasets and ListTables.  I recommend that you read more about these functions here.

3 Quick Python Tips to Make Geoprocessing Easier

1. Ignoring escape characters
 
When you export a Python script from Model Builder, you have probably noticed that the “\” in the directory paths are modified in two ways, either “/” or “\\”.  This is because “\” is an escape character.  Basically, “\” tells Python to look at the following character and do something special.  For example “\n” tells Python to start a new line.  However, if you are like me, you usually copy and paste a file path and it is a hassle to have to change every occurrence of “\”.  Luckily there is away to tell Python to ignore the “\” in a string as an escape character.  You can do this by placing an “r” directly in front of the string.  So here are three ways to create the same string:
 
r"C:\Program Files\ArcGIS\ArcToolbox\Toolboxes"
"C:/Program Files/ArcGIS/ArcToolbox/Toolboxes"
"C:\\Program Files\\ArcGIS\\ArcToolbox\\Toolboxes"


2. Getting the directory location of the current script

When creating batch scripts that you will reuse frequently for files in various locations, it can be time consuming to modify the file directory before running the script every time.  One method of making this step easier is by placing the script in the same directory as the files you want to process.  To do this, you can have the script use the directory it is located in using the getcwd function.  This function returns the directory that the script is located in.  First import the os module and then call the getcwd function:

import os

directory = os.getcwd()


3. Create a data stamp

When processing file, you may want to create a date stamp on the end of the file name or write the date to a txt file.  Creating a date stamp is fairly easy First import the time module, then call the localtime function to return the current date and time on the system.  This example returns a string in the “MM_DD_YYYY format:

import time

t = time.localtime()

date = str(t[1]) + "_" + str(t[2]) + "_" + str(t[0])

Python Tip: Using ArcSDESQLExecute to access ArcSDE data faster

In a previous post titled “Using Python to access ArcSDE data faster with cx_Oracle” I described a Python module that allowed faster access to ArcSDE data stored in an Oracle database.  Another method exists that will allow you to access ArcSDE data with pure SQL queries using the ArcScripting geoprocessing libraries.  We can accomplish this by creating the ArcSDESQLExecute object.

To help demonstrate the capabilities of this function, we will create an example that queries the ArcSDE database and then uses some Python functionality to generate a CSV file from the results.  Start by importimg the necessary libraries and creating a geoprocessor object:

import arcgisscripting, os

gp = arcgisscripting.create(9.3)

Next create the ArcSDESQLExecute object.  The second parameter is the path to the SDE connection file:

sdeConn = gp.CreateObject("ARCSDESQLEXECUTE", "Database Connections\\GIS.sde")

Now let’s create an SQL statement and execute it.  Note:  This function will execute any valid SQL statement that you have permission to perform, including Updates and Deletes.  This is an excellent way of performing fast attribution changes.  However, use caution when modifying data:

sql = "SELECT OBJECTID, ROADNAME FROM ROADSEGMENT"

sdeReturn = sdeConn.Execute(sql)

The next section of code will generate a CSV file so that the output can easily be opened in Excel or Access.  You can modify it to any type of TXT file you wish.

file = open("output.csv", 'w')

for row in sdeReturn:

    line = ""

    for i in row:

        line += str(i) + ","

    file.write(line + "\n")

Finally, close the file and delete the ArcSDESQLExecute object:

file.close()

del sdeReturn

You will see that this function performs SQL queries much faster than the other geoprocessing tools because it uses the RDBMS to execute them.  ArcSDESQLExecute can even rollback transactions.  For more information see the ESRI documentation.

Using GDAL with ArcGIS Explorer

ArcGIS Explorer has extremely limited functionality right out of the box. If you are looking to do anything other than simply viewing data, you may find it very difficult with the ArcGIS Explorer api and standard .NET libraries. Fortunately, there are many open source GIS libraries that can be used to fill the gaps where the api is lacking. One such library is GDAL / OGR. These libraries can be found in the FWTools software package and are used in various commercial applications.

Much of the ArcGIS Explorer functionality is provided by these exact libraries. Therein lies our problem. The fact that the libraries are used in the application is not a problem in itself. The problem typically surfaces when you download the latest version of the library and attempt to use with Explorer. You may run into all sorts of odd problems, with the most common being a "dll not found" error. This error can be very misleading. The first reaction might be to see if the dll actually exists. If you check, you will find that it DOES exist, and it is in a location where the application should be able to find it. So why does Explorer give a dll not found error? Welcome to dll hell. For many a COM developer, this is familiar territory, but not so for .NET developers. In .NET, dll hell has been all but eradicated.

For those unfamiliar with dll hell, the short explination goes something like this: windows has a specific order in which it looks for dll's. It typically starts with the directory containing the application, after which it will search through each of the directories in the path variable. During this sequence, it will load the very first dll it finds with a matching name. "So what happens when the dll it finds is not the one my application is expecting?" you ask. This, my friends, is dll hell. I cannot tell you specifically what will happen in your case, but chances are it will not be good.

So why are we experiencing it here? Well, GDAL is not a native .NET library. It is written in C/C++ with .NET wrappers built on top. To some degree, the "dll not found" error is accurate, because it did not find the version of the dll it was searching for. So how do we fix that? Well, the simple answer is to simply use the same version of GDAL that is used by the main application. What is good about this is that you will not even need to include the dlls with your deployment, since they already exist within the application. You will have to include any of the .NET wrappers you use, however.

One roadblock that you may run into is finding the right version of GDAL. In the case of ArcGIS Explorer 10, GDAL 1.6.1 is used. Tracking down the right files can prove troublesome, since the version you need may not exist in binary form, which means you may have to download the source and compile it yourself. Since GDAL is typically built using nMake files, this can be quite a chore for the inexperienced. Fortunately, I have already done this for you, and I'm providing the .NET dlls for your convenience here.

If the version of GDAL that is included with the version of the application you have is different than the one posted here, you can find the instructions on how to build from source here.

Using GDAL with ArcGIS Explorer, you can potentially add on a mountain of functionality that does not exist natively within the application. For those who need some additional functionality but cannot afford to pony up the dough for a full blown ArcGIS license, this may be the solution you are looking for.

Using ArcObjects for coordinate transformations in Python

Some of you may find yourselves in situations where you may need to develop in Python but are limited by the existing geoprocessing methods available.  Luckily, there is a method of accessing the ArcObject COM libraries through Python.  The comtypes package allows you use all of the methods available to .NET developers.  The presentation Using ArcObjects in Python gives an excellent overview of how to accomplish this.

I’m going to walk you through a specific example using comtypes.  In this example, we will perform a coordinate conversion on a point from a Geographic Coordinate System to a Projected Coordinate System.  To get started, download and install comtypes.  Once this is complete it’s time to start coding.  We’ll start off by defining four functions.  These will make more sense later.  First get the COM library path:

def GetLibPath():

   import _winreg

   keyESRI = 
      _winreg.OpenKey(_winreg.HKEY_LOCAL_MACHINE,

                        "SOFTWARE\\ESRI\\ArcGIS")

   return _winreg.QueryValueEx(keyESRI,  
          "InstallDir")[0] + "com\\"

Next we need to create a procedure to get the ArcObjects module:

def GetModule(sModuleName):

   import comtypes

   import comtypes.client as comc

   sLibPath = GetLibPath()

   comc.GetModule(sLibPath + sModuleName)

Create a function that creates a new object:

def NewObj(MyClass, MyInterface):

   import comtypes

   try:

        ptr = comtypes.client.CreateObject(MyClass,
               interface=MyInterface)

        return ptr

   except:

        return None

Finally, create a function for casting objects:

def CType(obj, interface):

   try:

        newobj = obj.QueryInterface(interface)

        return newobj

   except:

        return None

Now we need to define the main function to perform the coordinate transformation:

def TransformCoords(x, y, gcs, pcs, trans, direction):

The x and y are the longitude and latitude of the point in decimal degrees.  In order to perform transformation you will need to look up some ESRI codes.  The list of geographic coordinate system codes are here: Page 1, Page 2, Page 3.  The projected coordinate systems are here: Page 1, Page 2, Page 3, Page 4.  And the transformations are here: Page 1, Page 2, Page 3.   Lastly, you’ll need the direction codes here.

For those familiar with writing ArcObjects in C# I will include the C# equivalent with each piece of code.  Now we need to get the Geometry module and import it:

   GetModule("esriGeometry.olb") 

   import comtypes.gen.esriGeometry as esriGeometry

C#:

using ESRI.ArcGIS.Geometry;

Next we create a new point object and assign the x and y values:

   point = NewObj(esriGeometry.Point,
          esriGeometry.IPoint)

   point.X = x

   point.Y = y

C#:

Point point = new Point();

point.X = x;

point.Y = y;

It’s time to start using the codes we looked up.  Let’s define the spatial reference for the geographic coordinate system:

   sRefEnv = 

       NewObj(esriGeometry.SpatialReferenceEnvironment,

          esriGeometry.ISpatialReferenceFactory)

   incomingCoordSystem =

   CType(sRefEnv.CreateGeographicCoordinateSystem(gcs),

      esriGeometry.ISpatialReference)
   point.SpatialReference = incomingCoordSystem

C#:
SpatialReferenceEnvironment sRefEnv = new  
   SpatialReferenceEnvironment();

ISpatialReference incomingCoordSystem = 

   sRefEnv.CreateGeographicCoordinateSystem(gcs);

point.SpatialReference = incomingCoordSystem;

Define the projected coordinate system:

   outgoingCoordSystem = 

   CType(sRefEnv.CreateProjectedCoordinateSystem(pcs), 

      esriGeometry.ISpatialReference)

C#:

ISpatialReference outgoingCoordSystem = 
sRefEnv.CreateProjectedCoordinateSystem(pcs);

Define the geographic transformation:

   geoTransformation = 

      CType(sRefEnv.CreateGeoTransformation(trans), 

         esriGeometry.IGeoTransformation)

C#:

IGeoTransformation geoTransformation = 
(IGeoTransformation)sRefEnv.CreateGeoTransformation(trans);

Re-project the coordinates and assign the new geometry to the point:

   geometry = CType(point, esriGeometry.IGeometry2)

   geometry.ProjectEx(outgoingCoordSystem, 
          direction,geoTransformation, False, 0, 0)

   point = CType(geometry, esriGeometry.IPoint)

       

C#:

IGeometry2 geometry = point as IGeometry2;

geometry.ProjectEx(outgoingCoordSystem, esriTransformDirection.esriTransformForward, geoTransformation, 
   false, 0, 0);

point = (Point)geometry;

Finally, assign the coordinates to an array and return them:

   points = [point.X, point.Y]

       

   return points

In the end you have an array of the transformed x and y of the original point.  Another method exists that can also be helpful called ctypes.  With ctypes, you can access C libraries much like comtypes.  An excellent example of transformation using ctypes can be found here.

Using Python to access ArcSDE data faster with cx_Oracle

Using Python and ArcObjects to access data can be painfully slow, especially with large amounts of data.  However, there is another method for accessing data from Oracle that will allow your scripts to run faster.  The Python extension module, cx_Oracle, allows you to access an Oracle database without using ArcObjects.  Using cx_Orcale you can quickly query and modify attributes for tables and feature classes in ArcSDE.

NOTE:  Do NOT attempt to modify (INSERT, UPDATE, DELETE, etc) a table or feature class that is registered as versioned.  The results are unpredictable at best and can result in data loss.  However, you can safely SELECT any table or feature class.

To start you’ll need to download the cx_Oracle module here.  Once you have it installed and are ready to write some code, begin by importing the cx_Oracle module:

import cx_Oracle

Next we need to connect to the Oracle database.  There are several ways to do this.  For a complete reference, see the cx_Oracle documentation.  In this example we first create a DSN:

dsn = cx_Oracle.makedsn("gisdb01.devfaction.com", 

      "1521", "gisdb")

In this example, gisdb01.devfaction.com is the host, 1521 is the port, and gisdb is the name of the database.  Once the DSN is created, we connect to the database using the DSN:

orcl = cx_Oracle.connect('', '', dsn)

Now that we are connected to the database, we need to create a cursor:

cursor = orcl.cursor()

The next step is to create a SQL statement to execute.  This statement can be any SQL syntax that Oracle supports.  We will start off with a simple SELECT statement:

sql = "SELECT id, name FROM DB.ROADS WHERE id < 10"

This statement will return two fields from the ROADS table where the ID is less than 10.  Now it’s time to execute the statement:

cursor.execute(sql)

Now that we have run a query, we need to use the data that was returned.  To this we can loop through each row in the cursor:

for row in cursor:

     print row[0] + ", " + row[1]

The results should look something like this:

1, Walnut

2, Oak

3, Spruce

When you are finished remember to delete the cursor and close your connection:

del cursor

orcl.close()

You can also delete and modify data using cx_Oracle.  It may be possible to INSERT new records but it is not recommended for ArcSDE objects.  Follow the same steps to create the connection and cursor then create a SQL statement and execute it.:

sql = "UPDATE DB.ROADS SET name = '1st' " +\

      "WHERE id = 3"

cursor.execute(sql)

You will need to commit your changes before closing the connection:

orcl.commit()

In addition to creating scripts that run faster, cx_Oracle can help users who are more familiar with writing SQL than using ArcObjects.  But I want to reiterate one more time that modifications should not be performed on feature classes or tables that are registered as versioned.

cx_Oracle consists of many more useful functions.  I encourage you to read through the documentation and see what else exists.

Development Faction Company Info

Established in 2009, Development Faction, LLC is the collective vision of several industry professionals highly skilled in the disciplines of Geospatial Information, Software Design & Development, Database Design / Management, Web Design & Implementation, and Project Management.

With a combined experience of over 20 years in the industry, Development Faction possesses the knowledge to evaluate, plan, and strategically implement custom tailored solutions to meet our customer’s needs.

Our unique business structure and well rounded skill set awards us the opportunity to handle projects of any size / complexity by eliminating the use of sub-consultants.

Our esteemed staff is well versed in using the industry standard commercial-off-the-shelf (COTS) software packages, developing custom software / applications, and open source software packages; therefore Development Faction can fulfill our customer’s needs in the most cost efficient manner.

Development Faction is an official Microsoft® BizSpark™ software development start-up firm.

For more information on Development Faction, visit our website:

http://www.devfaction.com

or contact us:

info@devfaction.com