Referenced Mosaic Datasets Permissions on SDE

While I’m enjoying the functionality of the new Referenced Mosaic Dataset that have been introduced in ArcGIS10, something that I stumbled over recently was administering the privileges to a referenced mosaic dataset.

A referenced mosaic dataset is a raster datatype that uses a mosaic dataset as a base.  For example, we loaded a series of DEMs into a mosaic dataset and then created a referenced mosaic dataset that does the meters to feet conversion for us.  This saved us from having to process each DEM before loading it.  We also created a referenced mosaic dataset hillshade from the original meter data.

The problem I came across, and this was more my oversight than anything, but I had a heck of a time trying to grant privileges to the referenced mosaic dataset.  The solution was actually very simple–I needed to also grant access to the mosaic dataset that was being used as the base.

That makes some sense, you should not be able to grant access to a dataset (the referenced mosaic dataset) that would inadvertently be exposing data (the base mosaic dataset) that a user does not have access to.  This, however, conflicts a bit in that the permissions on individual rasters that make up a mosaic raster dataset are ignored.

ArcMap Field Calculator: Calculating Geometry using arcpy

One of the things I had not gotten around to doing in ArcGIS 10 was figure out how to directly manipulate the geometry of a record using the Field Calculator.  When I stumbled upon a bug in the way the Extract Values to Points tool handles Null geometries, I figured it was time to figure it out.  Setting the X, Y to 0,0 was sufficient for my needs.

I set the Parser to Python and the formula was simple once I figured out the syntax:

pPoint = arcpy.Point(0,0)

Then, to extend my knowledge, I wanted to see how to calculate the geometry based off of two fields (X and Y).   The only real trick is knowing the bracket field names using exclamation points instead of brackets:


So turns out everything it pretty easy and straight-forward.  Whew!

Extract Values to Points (Spatial Analyst) Bug

One of the Spatial Analyst tools we often use in ArcGIS is the “Extract Values to Points” tool.  This allows us to take a point file (well locations in our case) and attach a value (elevations) from a raster image (a DEM) to each point.

Today I was running it for the first time against an Image Service we recently published and I received a warning message,”WARNING 000957: Skipping feature(s) because of NULL or EMPTY geometry”.  But the script seemed to run and the final results said “Succeeded” so I thought it was probably fine.  But as I double-checked, I realized the results were wonky.

Turns out that I had two records with Null geometry in my point file of 397 records.  These two records threw the above error but actually had a value in the [Rastervalu] field.  Turns out all 397 records had values.  These two records were consecutive–let’s say the 100th and 101st records in my shapefile.  What happened is record 100 got the value for record 102, record 101 get the value for 103, record 102 (which has valid geometry) had the value for 104.  This pattern, each record having the value for the record 2 place after it, continued until record 396 which had the value for record 397.  Record 397 also had the value for 397.  So the final three records all had the value for the final record.

What I would have expected would be for the two records with Null Geometry to have null values in the [Rastervalu] field and the rest of the records to have the correct values.  Despite the warning, it is very misleading for all the records to end up with a value.

I have a simplified example below.  I made a point shapefile with four records.  The first, third, and fourth  records have valid geometries; the second has Null geometry.  The second record ends up with the value for the third record.  The third record, has the value for the fourth.  The fourth record being the last record, ends up with the last valid value, which was its own.

The results that I would have hoped for would be for the third record to have a Null value.

The way I envision what is occurring behind the scenes is this:  the process makes a list (more of a stack in programming terms) of result values as it processed the points but just assumes that every record will return a value so it does not track which value goes with which shape.

When it reached the two null geometries, it threw an error but continued on.  It did not add a value for these records to the stack of values–when it comes across records with valid geometry but do not intersect the raster it adds a psuedo-null value of (-9999) to the stack.  After it processed all the records it had 395 values in the stack.  It then went, one-by-one through the stack and populated the records in the output shapefile, the first record got the first value in the stack, the second record got the second value, the 100th record got the 100th value (which came from the location of the 102nd record) and so on.  At the end, the final two records received the last valid value.

This final behavior–using the last valid value–corresponds a bit to a behavior we’ve seen with ArcObjects in general.  When iterating through a table, if a field is Null for a specific row, the value from the last non-Null value for that field is often returned.

I’m in the process of submitting a bug to ESRI.  I’m not sure if this existed prior to ArcGIS 10.0 (I’m guessing it did) or if it occurs in other processes (I’m guessing it does).  I did find out that the “Extract Multi Values to Points” works as expected.  I’m guessing it is because unlike the “Extract Values to Points” which creates a new shapefile, this tool appends fields to the existing shapefile and presumably processes records one-by-one without putting the results in a virtual stack.  The “Extract Multi Values to Points” tool also does not throw any warnings.

Instantiating Add-In Objects at Load Time

In migrating a toolbar consisting of a button and a couple of tools for use in ArcMap 10, I decided to take advantage of the ease of deployment enabled by add-ins which was introduced in 10.0.  So far, I’m loving the functionality.

One thing, however, that I have to figure out is that the controls are not instantiated until they are clicked on.  One of the results is that the controls, by default, are enabled.  This is not the functionality I wanted.

I found the solution in ESRI’s topic, Advanced add-in concepts (ArcObjects .NET 10 SDK) in the Delayed loading section:

By default, the assemblies associated with add-in buttons and tools are not loaded until the corresponding item on a toolbar or menu is clicked by the user. This behavior helps conserve application memory and other resources. Since the enabled state of an add-in button or tool is controlled by the OnUpdate method within code, the button or tool will initially appear enabled. If you need tighter control over the initial enabled state of a button or tool, you need to override the default behavior and force the item to load at startup by setting the onDemand Extensible Markup Language (XML) attribute to false.”

The verbiage, “setting the onDemand Extensible Markup Language (XML) attribute to false” was a bit non-specific to me.  I guessed right, however, that they were referring to the control definiton in the Command section of Config.esriaddinx.  In the example below, I set the onDemand attribute to false by adding the  ‘onDemand=”false”‘ tag and the control did, in fact, get instantiated at load time, giving me the ability to disable it.

<Tool id="MGS_QDIAddin_QDIAddTool" class="QDIAddTool" message="QDI Add New Location." caption="QDI Add Tool" tip="QDI Add Tool." category="QDI Add-In Controls" image="ImagesQDIAddTool.png" onDemand="false"/>

Loading Tiled, Same-Name Data in Batch Mode.

I have been loading existing raster data into a geodatabase to be included in a new Mosaic Dataset–a very cool and useful addition to ArcGIS 10. The most time-consuming part of the process for the human (at least this human) has been getting the names of the rasters right.

Our existing data is organized by tiles with the directory name representing the tile name and then the data within each tile directory having the same name.

For example:

This makes batch loading the data less efficient because I end up having to rename the data or else end up with a series of feature classes named parcels, parcels_2, parcels_n.

So I hacked out a quick script that takes an input raster and figures out the final name I want it to have based of the directory name.

First, I used the Copy Raster (In ArcToolbox: Data Management-Raster-Raster Dataset-Copy Raster) and copied on sample to my geodatabase.

Then, I went to the Results Tab (Select Geoprocessing from the Menubar, Geoprocessing-Results) and right-clicked on the Copy Raster result and selected “Copy as Python Snippet”.

I then created a new python script and pasted the one line.

I added some imports, accepted a parameter, some string manipulation, and some result outputs and I had a quick & easy script. In added the script in ArcToolbox and now I can right-click on it and run it in Batch mode. I do a quick search in Windows Explorer to get all the rasters I want to run it on and select & drag them to my ArcToolbox Batch Dialog.

Actual code can be downloaded HERE and you don’t need to worry about WordPress messing up the spacing.

import arcpy
import os, sys

inRaster = sys.argv[1] 
basedir = os.path.basename(os.path.dirname(inRaster)).lower()
outRaster = "Database Connections/mgs_lidar.lidar.sde/mgs_lidar.lidar."+basedir

def printit(inMessage):
    print inMessage
if not (arcpy.Exists(outRaster)):
    printit ("Importing: "+basedir)
    printit ("Skipping: "+basedir+" because it already exists!")

ESRI Font Samples.pdf (application/pdf Object)

Whenever I want to skim through ESRI’s fonts to find some symbology I need for a specific purpose, I go through some laborious process to skim the fonts searching for an adequate symbol.

However, in the ESRI Mapping Center blog post, user use2b311 post a link to a pre-made pdf showing ESRI Font Samples.pdf (application/pdf Object).

Back when ESRI use to actually print hard-copy manuals, there were similar diagrams available but either I haven’t searched enough of they don’t exist in the digital help systems so I was happy to stumble upon this.

A tangent discovery in checking this out was dropbox, a free (with 2 Gig limit) online file sharing/online backup site–something I’ve had a need for but haven’t gotten around to researching. I have not tried them but may, just to see how well it works.

Checking to see if a Field Index Exists Using Arcpy (ArGIS 10.0)

Updating some python code from 9.3 that using geoprocessing  to 10.0 using arcpy and the first real object I’ve had to change relates to detecting whether or not an index exists on a table.

I previously posted code using a 9.3 geoprocessing commands, the core of it being:

indexList = gp.listindexes(tablename)
for iIndex in indexList:
    if (iIndex.Name == indexname):
       return True
return False

With arcpy, ESRI has gone back to using the Describe methodology.  Way back when GISers use to type most of their commands into their wood-burning computers, the command “Describe” was used to retrieve information about a coverage, grid, or other data set in our fancy AMLs (or SMLs).  The snippet below shows a function for checking to see if a table has an index with a specified named.

def indexExists(tablename,indexname):

 if not arcpy.Exists(tablename):
  return False

 tabledescription = arcpy.Describe(tablename)

 for iIndex in tabledescription.indexes:
  if (iIndex.Name == indexname):
   return True

 return False

Multiple geodatabases and ArcSDE services from one PostgreSQL server.

To better organize our ArcSDE data, we wanted to create multiple geodatabases and multiple ArcSDE services using one PostgreSQL database cluster (a cluster containing 1 machine at this point).  A side question is why can’t tables and raster be placed in Feature Datasets?  This wouldn’t be an end-all solution for what we want to do but there are some messy consequences of this limitation.

ESRI has instructions on Setting up multiple geodatabases in one PostgreSQL database cluster on Windows which was helpful but we repeatedly got an “The ArcSDE Repository was unsuccessfully completed.  Would you like to view this error?” error during the Repository Setup step, step two of four, of the ArcSDE Post Installation process. The odd thing, however, was that the wise_err.log was blank so I had no clue on why it was failing.  After much Googling and head-banging, I came across a post from Kim Doty on ESRI’s forums reporting the same problem but they were able to create a service by just continuing through the process .  Figuring I had nothing to lose, I thought I might as well try to go through the entire ArcSDE Post Installation process and see what happened.  Although I received blank error messages, I was able to successfully create the second ArcSDE instance.

I will touch on some of the highlights of the process.  In my case, I was creating a database & service with these parameters:

  • Database name: mgsgdb_lidar
  • Service name:  mgs1_sde
  • Tablespace folder: D:mgsdb1dmgsgdb_lidar

Following ESRI’s instructions, I first made custom copies of dbinit.sde, dbtune.sde, and giomgr.defs that contain the service name (mgs1_sde) within the file name.  These copies were named dbinit_mgs1_sde.sde, dbtune_mgs1_sde.sde, and giomgr_mgs1_sde.defs.

When I have trouble with the ArcSDE Post Installation, I like to go through the four steps individually so I select a “Custom” install.

The four steps making up the ArcSDE Post Installation process are shown on the next dialog.

I was able to complete the first step, Define SDE User Environment, without any problems.

Make sure to set your database name, default tablespace, and tablespace folders are distinct from your first installation.

During the second step of the ArcSDE Post Installation process, Repository Setup, make sure to use the custom files you made earlier when you come to the ArcSDE configuratino files dialog.

Later in the Repository Setup, make sure to use the correct database name:

This is the first error message I received in the process (still during Repository Setup):

Clicking “Yes”, however, shows an empty wise_err.log file:

After viewing this, I canceled out of the Repository Setup step.

Taking a look in pgAdmin III, however, I can see that both my database (mgsgdb_lidar) and tablespace (mgsgdb_lidar) were created:

I received the same error message during the third step, Authorize ArcSDE, as in the second step:

But running the fourth step, Create ArcSDE Service, resulted in this sweet message:

After editing my pg_hba.conf and opening the port in my firewall, the service was created, running, and visible.  So far, it seems to be fully functional and without problems.

Walkthrough: Building custom UI elements using add-ins (ArcObjects .NET 10 SDK)

I was working my way through this ESRI Walkthrough: Building custom UI elements using add-ins (ArcObjects .NET 10 SDK).  And came across a couple minor errors that I had to correct during the process.

First, while implementing the OnClick() code for ZoomToLayer.vb, Visual Studio gave me a “Name ‘ArcMap’ is not declared.” error.

In the walk-through, they mention that the ArcMap method of your class.  For me, however, it appeared under the .My method.  Not sure if this is something specific to my set-up or, as I’m guessing, something that got changed after the first documentation was created and the final libraries published.

The fix is just adding  “My.” to the namespace in this line:

ZoomToActiveLayerInTOC(TryCast(ArcMap.Application.Document, IMxDocument))

To get this:

ZoomToActiveLayerInTOC(TryCast(My.ArcMap.Application.Document, IMxDocument))

When I added the code for AddGraphics.vb, I got 8 errors.  There was essentially two errors, repeated four times.  I took a screen shot after fixing the first error pair:

The fixes in this case was also to use the complete name space path.  Examples:

Change this:

(geometry.GeometryType) = esriGeometryType.esriGeometryPoint Then

To this:

If (geometry.GeometryType) = ESRI.ArcGIS.Geometry.esriGeometryType.esriGeometryPoint Then)

And change this:

simpleMarkerSymbol.Style = esriSimpleMarkerStyle.esriSMSCircle

To this:

simpleMarkerSymbol.Style = ESRI.ArcGIS.Display.esriSimpleMarkerStyle.esriSMSCircle

Overall, the walk-through is very well done, just a couple minor tweaks.  I am now working my way through modifying an existing solution–one that included seven projects–to see if I can create an ArcGIS 10 Add-In.

ArcGIS 9.3.1 to ArcSDE 10 Connection Error

We finally installed an instance of ArcSDE 10 today.  My first attempt at connecting in ArcCatalog 9.3.1 failed with the following error:

Failed to connect to the specified server.

This release of the GeoDatabase is either invalid or out of date. [Please run the

ArcSDE setup utility using the -o install option.]

DBMS table not found[sde.sde.GDB_Release]


Turns out the solution was simple, this article points out that Service pack 2 is required.  After updating my ArcGIS, the problem was solved:

I was the proud, new recipient of a non-misleading error message.  After some minor head-slapping, I realized that 9.3.1 and older versions of ArcGIS are not able to connect to ArcSDE 10.  Doh!  My bad–I should have read ESRI’s clear information on this topic.  At least I never actually tried to install ArcSDe with the -o option as the first error message instructed me to do.